Downloading a Directory Tree with Ftplib

Python - download all folders, subfolders, and files with the python ftplib module

The short solution:
You could possibly just run: "wget -r ftp://username:password@ftp.debian.org/debian/*" to get all the files under the debian directory.
Then you can process the files in python.

The long solution:
You can go over every directory listing by using ftplib, getting a directory listing parsing it and then getting every file and recursing into directories.
If you search the web you'd find previous posts on stackoverlow which solve this issue

Using Python's ftplib to get a directory listing, portably

Try using ftp.nlst(dir).

However, note that if the folder is empty, it might throw an error:

files = []

try:
    files = ftp.nlst()
except ftplib.error_perm as resp:
    if str(resp) == "550 No files found":
        print "No files in this directory"
    else:
        raise

for f in files:
    print f

Download a whole directory (containing files and subdirectories)

The directory creation seems to be similar to this question that was already asked besides the change due to Windows format.

mkdir -p functionality in Python

How to run os.mkdir() with -p option in Python?

As a result, I would suggest putting in the mkdir_p function shown in the accepted answers to those questions and then seeing if the windows will create the appropriate path

os.mkdir(destination[0:len(destination)-1]+path.replace("/", "\\"))

then becomes

newpath = destination[0:len(destination)-1]+path.replace("/", "\\")
mkdir_p(newpath)

This uses the os.makedirs(path) method in order to get the full path. You could also just replace os.mkdir() with os.makedirs()

Note that if you use the replacement path in a number of places, then just keep using the newpath variable. That might be easier in the rest of your code.

When uploading recursively all files to FTP using ftplib, only the first directory is successfully transferred

You enter the remote folders here:

ftp_server.cwd(filename)

But you never leave them.

---

Either:

• Leave the folder, once you download its contents:

  ftp_server.cwd(filename)
  copy_all_files(ftp_server, os.path.join(path, filename))
  ftp_server.cwd("..")
  

• Or use absolute paths when entering the folders. That way the cwd will work, no matter there you are at that particular moment. Like here:
  
  Downloading a directory tree with ftplib

  In a broad sense, your question is actually a duplicate of the above one – Why are you trying to implement something, for which there's already a working solution?

Recursive file list with FTP

I could do it by entering manually each dir, and then redo a msld()

And that's the correct solution. FTP protocol does not have any better standard way to retrieve recursive listing. So there's no space for optimization (only by parallelizing the operation). See also Downloading a directory tree with ftplib.

Some FTP servers support non-standard -R switch with some file listing commands (not sure about MLSD). So if you are willing to rely on a non-standard functionality and your particular server supports it, you can optimize your code this way. See also Getting all FTP directory/file listings recursively in one call.

For SFTP, see Recursive SFTP listdir in Python?

How to store a directory structure from an ftp site?

No native datatype that I'm aware of. Note that a directory structure is a pretty simple tree structure though. If you need more than just a simple tree listing, I'd recommend rolling your own Directory and File objects. Allow your Directory object to act like a dict and you'll be able to do things like root['etc']['ssh']['config'].

If you just need a simple tree structure, have you considered just using nested dicts?

root = {
    'etc': {
        'passwd': None, 
        'shadow': None, 
        'ssh': {
            'config': None}
    }, 
    'lib': {
    }
}

I'm using None as the data value for a leaf node, but you could certainly store file metadata there instead if you like. Navigating the tree structure is very simple. A listing of /etc is just root['etc'].keys().

Finally, pet peeve time. "I am always looking for more pythonic ways of doing stuff" What does that mean? It's perfectly reasonable to look for ideas/advice/guidance, but at the end of the day, the guy with the working code wins. In this case, the "pythonic way of doing stuff" is to just do it. In Python, of course :)

---

Related Topics