Using Pyinotify to Watch for File Creation, But Waiting for It to Be Completely Written to Disk

Using pyinotify to watch for file creation, but waiting for it to be completely written to disk

Have pyinotify react to IN_CLOSE_WRITE events:

wm.add_watch(watched_dir, pyinotify.IN_CLOSE_WRITE, proc_fun=MyProcessEvent())

This is from man 5 incrontab, but it applies equally well to pyinotify:

   IN_ACCESS           File was accessed (read) (*)
   IN_ATTRIB           Metadata changed (permissions, timestamps, extended attributes, etc.) (*)
   IN_CLOSE_WRITE      File opened for writing was closed (*)
   IN_CLOSE_NOWRITE    File not opened for writing was closed (*)
   IN_CREATE           File/directory created in watched directory (*)
   IN_DELETE           File/directory deleted from watched directory (*)
   IN_DELETE_SELF           Watched file/directory was itself deleted
   IN_MODIFY           File was modified (*)
   IN_MOVE_SELF        Watched file/directory was itself moved
   IN_MOVED_FROM       File moved out of watched directory (*)
   IN_MOVED_TO         File moved into watched directory (*)
   IN_OPEN             File was opened (*)

Check and wait until a file exists to read it

A simple implementation could be:

import os.path
import time

while not os.path.exists(file_path):
    time.sleep(1)

if os.path.isfile(file_path):
    # read file
else:
    raise ValueError("%s isn't a file!" % file_path)

You wait a certain amount of time after each check, and then read the file when the path exists. The script can be stopped with the KeyboardInterruption exception if the file is never created. You should also check if the path is a file after, to avoid some unwanted exceptions.

Webdriver open a file as soon as it finishes downloading

You could try hooking the file up to a file object as it downloads to use it like a stream buffer, polling it as it downloads to get the data you need, monitoring for the download completion yourself directly (either by waiting for the file to be of the expected size or by assuming it is complete if there has been no new data added for a certain amount of time).

Edit:

You could try to look at the download tracking db in the profile folder as referenced here. Looks like you can wait for your file to have status 1.

Detect File Change Without Polling

For linux, there is pyinotify.

From the homepage:

Pyinotify is a Python module for
monitoring filesystems changes.
Pyinotify relies on a Linux Kernel
feature (merged in kernel 2.6.13)
called inotify. inotify is an
event-driven notifier, its
notifications are exported from kernel
space to user space through three
system calls. pyinotify binds these
system calls and provides an
implementation on top of them offering
a generic and abstract way to
manipulate those functionalities.

Thus it is obviously not cross-platform and relies on a new enough kernel version. However, as far as I can see, requiring kernel support would be true about any non-polling mechanism.

---

Related Topics