C# Httpwebrequest Command to Get Directory Listing

C# HttpWebRequest command to get directory listing

A few important considerations before the code:

The HTTP Server has to be configured to allow directories listing for the directories you want;
Because directory listings are normal HTML pages there is no standard that defines the format of a directory listing;
Due to consideration 2 you are in the land where you have to put specific code for each server.

My choice is to use regular expressions. This allows for rapid parsing and customization. You can get specific regular expressions pattern per site and that way you have a very modular approach. Use an external source for mapping URL to regular expression patterns if you plan to enhance the parsing module with new sites support without changing the source code.

Example to print directory listing from http://www.ibiblio.org/pub/

namespace Example
{
    using System;
    using System.Net;
    using System.IO;
    using System.Text.RegularExpressions;

    public class MyExample
    {
        public static string GetDirectoryListingRegexForUrl(string url)
        {
            if (url.Equals("http://www.ibiblio.org/pub/"))
            {
                return "<a href=\".*\">(?<name>.*)</a>";
            }
            throw new NotSupportedException();
        }
        public static void Main(String[] args)
        {
            string url = "http://www.ibiblio.org/pub/";
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
            using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
            {
                using (StreamReader reader = new StreamReader(response.GetResponseStream()))
                {
                    string html = reader.ReadToEnd();
                    Regex regex = new Regex(GetDirectoryListingRegexForUrl(url));
                    MatchCollection matches = regex.Matches(html);
                    if (matches.Count > 0)
                    {
                        foreach (Match match in matches)
                        {
                            if (match.Success)
                            {
                                Console.WriteLine(match.Groups["name"]);
                            }
                        }
                    }
                }
            }

            Console.ReadLine();
        }
    }
}

C# HttpWebRequest Directory Listing using Regex

Expand your regex to include the associated date text, and use named capture groups:

var regex = new Regex(@"(?<date>\d{1,2}/\d{1,2}/\d{4})[^""]*\""(?<path>[^""]*)\""");
...
foreach (var match in regex.Matches(html)) {
    var date = DateTime.ParseExact(match.Groups["date"].Value, @"M\/d\/yyyy", null);
    var path = match.Groups["path"].Value;
    ...
}

How to Get Files From A Directory That's Open for Directory Listing?

This code will help you get the html

             WebRequest request = WebRequest.Create("http://yourwebsite/yourwebdirectory/");
             var webResponse=request.GetResponse();
             Stream dataStream = webResponse.GetResponseStream(); 
             StreamReader reader = new StreamReader(dataStream); 
             string responseFromServer = reader.ReadToEnd(); 
             Console.WriteLine(responseFromServer); 
             reader.Close();
             webResponse.Close();

Once you get the html you can parse it to get the file names

I am trying to list all the files present in the ftp server using c# winforms application

I have written the related code in the msdn forum like: download each file in folder.

You can use the FtplistFile method to get a list of all files in the ftp server.

Here is a code example I modified.

private void button1_Click(object sender, EventArgs e)
    {
        string uri = "ftp://xxx.x.xx.x/";
        string username = "USERNAME";
        string password = "PASSWORD";
        ListFtpDirectory(uri, new NetworkCredential(username, password));

    }

    static void ListFtpDirectory(string url, NetworkCredential credentials)
    {
        WebRequest listRequest = WebRequest.Create(url);
        listRequest.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
        listRequest.Credentials = credentials;

        List<string> lines = new List<string>();

        using (WebResponse listResponse = listRequest.GetResponse())
        using (Stream listStream = listResponse.GetResponseStream())
        using (StreamReader listReader = new StreamReader(listStream))
        {
            while (!listReader.EndOfStream)
            {
                string line = listReader.ReadLine();
                lines.Add(line);
            }
        }

        foreach (string line in lines)
        {
            string[] tokens =
                line.Split(new[] { ' ' }, 9, StringSplitOptions.RemoveEmptyEntries);
            string name = tokens[3];
            string permissions = tokens[2];

            if (permissions == "<DIR>")
            {
                Console.WriteLine($"Directory {name}");
                string fileUrl = url + name;
                ListFtpDirectory(fileUrl + "/", credentials);
            }
            else
            {
                Console.WriteLine(url+name);
                Console.WriteLine($"File {name}");
            }
        }
    }

Get file name:

Sample Image

Get file path in ftp:

Sample Image

How to List Directory Contents with FTP in C#?

Try this:

FtpWebRequest ftpRequest = (FtpWebRequest)WebRequest.Create(uri);
ftpRequest.Credentials =new NetworkCredential("anonymous","janeDoe@contoso.com");
ftpRequest.Method = WebRequestMethods.Ftp.ListDirectory;
FtpWebResponse response = (FtpWebResponse)ftpRequest.GetResponse();
StreamReader streamReader = new StreamReader(response.GetResponseStream());

List<string> directories = new List<string>();

string line = streamReader.ReadLine();
while (!string.IsNullOrEmpty(line))
{
    directories.Add(line);
    line = streamReader.ReadLine();
}

streamReader.Close();

It gave me a list of directories... all listed in the directories string list... tell me if that is what you needed

List all files from a URL in a Console Application

you cannot use the Directory class to list files of a Web directory and also the Server has to be configured to allow Directories/Files listing

What you should do is a web request the return the file list.

Have a look here for more info

Listing server folders and sub folders using C# FtpWebRequest

I'm doing similar things, but instead of using StreamReader.ReadLine() for each one, I get it all at once with StreamReader.ReadToEnd(). You don't need ReadLine() to just get a list of directories. Below is my entire code (Entire howto is explained in this tutorial):

        FtpWebRequest request=(FtpWebRequest)FtpWebRequest.Create(path);
        request.Method=WebRequestMethods.Ftp.ListDirectoryDetails;
        List<ftpinfo> files=new List<ftpinfo>();

        //request.Proxy = System.Net.WebProxy.GetDefaultProxy();
        //request.Proxy.Credentials = CredentialCache.DefaultNetworkCredentials;
        request.Credentials = new NetworkCredential(_username, _password);
        Stream rs=(Stream)request.GetResponse().GetResponseStream();

        OnStatusChange("CONNECTED: " + path, 0, 0);

        StreamReader sr = new StreamReader(rs);
        string strList = sr.ReadToEnd();
        string[] lines=null;

        if (strList.Contains("\r\n"))
        {
            lines=strList.Split(new string[] {"\r\n"},StringSplitOptions.None);
        }
        else if (strList.Contains("\n"))
        {
            lines=strList.Split(new string[] {"\n"},StringSplitOptions.None);
        }

        //now decode this string array

        if (lines==null || lines.Length == 0)
            return null;

        foreach(string line in lines)
        {
            if (line.Length==0)
                continue;
            //parse line
            Match m= GetMatchingRegex(line);
            if (m==null) {
                //failed
                throw new ApplicationException("Unable to parse line: " + line);
            }

            ftpinfo item=new ftpinfo();
            item.filename = m.Groups["name"].Value.Trim('\r');
            item.path = path;
            item.size = Convert.ToInt64(m.Groups["size"].Value);
            item.permission = m.Groups["permission"].Value;
            string _dir = m.Groups["dir"].Value;
            if(_dir.Length>0  && _dir != "-")
            {
                item.fileType = directoryEntryTypes.directory;
            } 
            else
            {
                item.fileType = directoryEntryTypes.file;
            }

            try
            {
                item.fileDateTime = DateTime.Parse(m.Groups["timestamp"].Value);
            }
            catch
            {
                item.fileDateTime = DateTime.MinValue; //null;
            }

            files.Add(item);
        }

        return files;

List names of files in FTP directory and its subdirectories

It is s not easy to implement this without any external library. Unfortunately, neither the .NET Framework nor PowerShell have any explicit support for recursively listing files in an FTP directory.

You have to implement that yourself:

List the remote directory
Iterate the entries, recursing into subdirectories - listing them again, etc.

Tricky part is to identify files from subdirectories. There's no way to do that in a portable way with the .NET Framework (FtpWebRequest). The .NET Framework unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol. See also Checking if object on FTP server is file or directory.

Your options are:

Do an operation on a file name that is certain to fail for file and succeeds for directories (or vice versa). I.e. you can try to download the "name".
You may be lucky and in your specific case, you can tell a file from a directory by a file name (i.e. all your files have an extension, while subdirectories do not)
You use a long directory listing (LIST command = ListDirectoryDetails method) and try to parse a server-specific listing. Many FTP servers use *nix-style listing, where you identify a directory by the d at the very beginning of the entry. But many servers use a different format. The following example uses this approach (assuming the *nix format)

static void ListFtpDirectory(string url, NetworkCredential credentials)
{
    WebRequest listRequest = WebRequest.Create(url);
    listRequest.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
    listRequest.Credentials = credentials;

    List<string> lines = new List<string>();

    using (WebResponse listResponse = listRequest.GetResponse())
    using (Stream listStream = listResponse.GetResponseStream())
    using (StreamReader listReader = new StreamReader(listStream))
    {
        while (!listReader.EndOfStream)
        {
            string line = listReader.ReadLine();
            lines.Add(line);
        }
    }

    foreach (string line in lines)
    {
        string[] tokens =
            line.Split(new[] { ' ' }, 9, StringSplitOptions.RemoveEmptyEntries);
        string name = tokens[8];
        string permissions = tokens[0];

        if (permissions[0] == 'd')
        {
            Console.WriteLine($"Directory {name}");

            string fileUrl = url + name;
            ListFtpDirectory(fileUrl + "/", credentials);
        }
        else
        {
            Console.WriteLine($"File {name}");
        }
    }
}

Use the function like:

NetworkCredential credentials = new NetworkCredential("user", "mypassword");
string url = "ftp://ftp.example.com/directory/to/list/";
ListFtpDirectory(url, credentials);

If you want to avoid troubles with parsing the server-specific directory listing formats, use a 3rd party library that supports the MLSD command and/or parsing various LIST listing formats.

For example with WinSCP .NET assembly you can list whole directory recursively with a single call to Session.EnumerateRemoteFiles:

// Setup session options
var sessionOptions = new SessionOptions
{
    Protocol = Protocol.Ftp,
    HostName = "ftp.example.com",
    UserName = "user",
    Password = "mypassword",
};

using (var session = new Session())
{
    // Connect
    session.Open(sessionOptions);

    // Enumerate files
    var options =
        EnumerationOptions.EnumerateDirectories |
        EnumerationOptions.AllDirectories;
    IEnumerable<RemoteFileInfo> fileInfos =
        session.EnumerateRemoteFiles("/directory/to/list", null, options);
    foreach (var fileInfo in fileInfos)
    {
        Console.WriteLine(fileInfo.FullName);
    }
}

Not only the code is simpler, more robust and platform-independent. It also makes all other file attributes (size, modification time, permissions, ownership) readily available via the RemoteFileInfo class.

Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.

(I'm the author of WinSCP)

C# Httpwebrequest Command to Get Directory Listing