LINUX: how to detect that ftp file upload is finished
I'd try using inotify, event code IN_CLOSE_WRITE.
How to detect that a file is being uploaded over FTP
There's no generic solution to this problem.
Some FTP servers lock the file being uploaded, preventing you from accessing it, while the file is still being uploaded. For example IIS FTP server does that. Most other FTP servers do not. See my answer at Prevent file from being accessed as it's being uploaded.
There are some common workarounds to the problem (originally posted in SFTP file lock mechanism, but relevant for the FTP too):
You can have the client upload a "done" file once the upload finishes. Make your automated system wait for the "done" file to appear.
You can have a dedicated "upload" folder and have the client (atomically) move the uploaded file to a "done" folder. Make your automated system look to the "done" folder only.
Have a file naming convention for files being uploaded (".filepart") and have the client (atomically) rename the file after upload to its final name. Make your automated system ignore the ".filepart" files.
See (my) article Locking files while uploading / Upload to temporary file name for an example of implementing this approach.
Also, some FTP servers have this functionality built-in. For example ProFTPD with its
HiddenStores
directive.A gross hack is to periodically check for file attributes (size and time) and consider the upload finished, if the attributes have not changed for some time interval.
You can also make use of the fact that some file formats have clear end-of-the-file marker (like XML or ZIP). So you know, that the file is incomplete.
Some FTP servers allow you to configure a hook to be called, when an upload is finished. You can make use of that. For example ProFTPD has a mod_exec
module (see the ExecOnCommand
directive).
How to know if file is complete on the server using FTP?
This is a very old and well-known problem.
There is no way to be absolutely certain a file being written by the FTP daemon is complete. It's even possible that the file transfer failed and then gets restarted and completed. You must poll the file's size and set a time limit, say 5 minutes. If the size does not change during that time you assume the file is complete.
If possible, the program that processes the file should be able to deal with partial files.
A much better alternative is rsync, which is much more robust and deterministic. It can even be configured (via command-line option) to write the data initially to a temporary location and move it to its final destination path upon successful completion. If the file exists where you expect it, then it is by definition complete.
FTP: How to know if a file on remote ftp server is complete
I use ftputil to implement this work-around:
- connect to ftp server
- list all files of the directory
- call stat() on each file
- wait N seconds
- For each file: call stat() again. If result is different, then skip this file, since it was modified during the last seconds.
- If stat() result is not different, then download the file.
This whole ftp-fetching is old and obsolete technology. I hope that the customer will use a modern http API the next time :-)
Detect SFTP file upload completed
For problem 1) you can use the inotify
API to watch the directory for changes. You don't say which programming language you use. In Perl you would use Linux::Inotify
, in Python pynotify
.
For problem 2), one way is to wait until the ssh session disconnects, another is to also use inotify
to watch for file close notifications.
How to detect whether file uploaded by FTP
make a file checker.sh , and run it in background ./checker.sh /dev/null 2>&1 & , or run it in screen
#!/bin/bash
COUNTER=0
while [ $COUNTER -lt 99999 ]; do
if [ -f "/path/to/your/file" ]; then
#do action here ,run a script a program etc #the file is there
fi
sleep 5
let COUNTER=COUNTER+1
done
Check status of ftp file transfer in python
I actually found a solution. I'm using proftpd in Ubuntu, and it have a configuration when can add a sufix and/or prefix to the filename when is still on transfer, and then rename it to the original name.
I just had to ignore all file with the prefix or sufix. With this I guarantee that I open the file when is complete.
How to determine whether a file is still being transferred via ftp
Our server admin suggested ftpwho, which outputs which files are currently transferred.
http://www.castaglia.org/proftpd/doc/ftpwho.html
So the solution is to parse the output of ftpwho to see if a file in the directory is being transferred.
Related Topics
Url Encoding a String in Bash Script
Why Is Crond Failing to Run a Non-Root Crontab on Alpine Linux
Getting Current Path in Variable and Using It
How to Pass a Value to a Builtin Linux Kernel Module at Boot Time
Installing Gnuplot 5.0 on Ubuntu
What Does -Prune Option in Find Do
How Do Locales Work in Linux/Posix and What Transformations Are Applied
Curl Simple File Upload - 417 Expectation Failed
How to Run a Script Every Time I Save a File in Linux
How to Grep for a Pattern in the Files in Tar Archive Without Filling Up Disk Space
Receiving Udp Broadcast Packets on Linux
Is It Ok (Performance-Wise) to Have Hundreds or Thousands of Files in the Same Linux Directory
/Usr/Bin/Perl: Bad Interpreter: Text File Busy
How Is Stack Size of Linux Process Related to Pthread, Fork and Exec
How to Send Multicast Packets via a Specfic Interface in Linux
How to Translate Linux Keycodes from /Dev/Input/Event* to Ascii in Perl
Is There Any Shortcut to Reference the Path of the First Argument in a Mv Command