How to Lock a Directory Between Python Processes in Linux

Python: Lock directory

I change your code a bit,add return self like most context manage do,then with dup(),the second context manage will fail.and the solution is simple,uncommentfcntl.flock(self.dir_fd,fcntl.LOCK_UN)

The mode used to open the file doesn't matter to flock.

and you cannot flock on NFS.

import os
import fcntl
import time
class LockDirectory(object):
    def __init__(self, directory):
        assert os.path.exists(directory)
        self.directory = directory

    def __enter__(self):
        self.dir_fd = os.open(self.directory, os.O_RDONLY)
        try:
            fcntl.flock(self.dir_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
        except IOError as ex:             
            raise Exception('Somebody else is locking %r - quitting.' % self.directory)
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        # fcntl.flock(self.dir_fd,fcntl.LOCK_UN)
        os.close(self.dir_fd)

def main():
    with LockDirectory("test") as lock:
        newfd = os.dup(lock.dir_fd)
    with LockDirectory("test") as lock2:
        pass

if __name__ == '__main__':
    main()

Python sharing a lock between processes

You can't pass normal multiprocessing.Lock objects to Pool methods, because they can't be pickled. There are two ways to get around this. One is to create Manager() and pass a Manager.Lock():

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    m = multiprocessing.Manager()
    l = m.Lock()
    func = partial(target, l)
    pool.map(func, iterable)
    pool.close()
    pool.join()

This is a little bit heavyweight, though; using a Manager requires spawning another process to host the Manager server. And all calls to acquire/release the lock have to be sent to that server via IPC.

The other option is to pass the regular multiprocessing.Lock() at Pool creation time, using the initializer kwarg. This will make your lock instance global in all the child workers:

def target(iterable_item):
    for item in items:
        # Do cool stuff
        if (... some condition here ...):
            lock.acquire()
            # Write to stdout or logfile, etc.
            lock.release()
def init(l):
    global lock
    lock = l

def main():
    iterable = [1, 2, 3, 4, 5]
    l = multiprocessing.Lock()
    pool = multiprocessing.Pool(initializer=init, initargs=(l,))
    pool.map(target, iterable)
    pool.close()
    pool.join()

The second solution has the side-effect of no longer requiring partial.

Locking a file in Python

Alright, so I ended up going with the code I wrote here, on my website ^{link is dead, view on archive.org} (also available on GitHub). I can use it in the following fashion:

from filelock import FileLock

with FileLock("myfile.txt"):
    # work with the file as it is now locked
    print("Lock acquired.")

How to properly implement a process based lock in Python?

You can create your own context manager with contextlib, and use fcntl to issue the locking calls. Note that these can be made non-blocking.

Both contextlib and fcntl are part of the standard library.

For you to experiment with stale locks, you can try launching the process twice and issuing a SIGKILL to one of the two — you should see the lock being released on the other process.

import fcntl
import contextlib

@contextlib.contextmanager
def lock(fname):
    with open(fname, "w") as f:
        print "Acquiring lock"
        fcntl.lockf(f, fcntl.LOCK_EX)
        print "Acquired lock"

        yield

        print "Releasing lock"
        fcntl.lockf(f, fcntl.LOCK_UN)
        print "Released lock"

if __name__ == "__main__":
    import os
    print "PID:", os.getpid()

    import time
    print "Starting"
    with lock("/tmp/lock-file"):
        time.sleep(100)
    print "Done"

System-wide mutex in Python on Linux

The "traditional" Unix answer is to use file locks. You can use lockf(3) to lock sections of a file so that other processes can't edit it; a very common abuse is to use this as a mutex between processes. The python equivalent is fcntl.lockf.

Traditionally you write the PID of the locking process into the lock file, so that deadlocks due to processes dying while holding the lock are identifiable and fixable.

This gets you what you want, since your lock is in a global namespace (the filesystem) and accessible to all processes. This approach also has the perk that non-Python programs can participate in your locking. The downside is that you need a place for this lock file to live; also, some filesystems don't actually lock correctly, so there's a risk that it will silently fail to achieve exclusion. You win some, you lose some.

How to create a system-wide file lock?

There is support in *NIX to apply locks to files (or parts of files). It requires the processes to co-operate with each other (advisory locking). Each process should always check if a given file or record is locked before accessing it. If this check is not performed there is no built in protection to avoid possible corruption. So in your case both processes have to use file locking for this to work.

It is good idea, if only documentary, to use LOCK_UN (unlock) explicitly, although locks are released automatically when a process exits or when the file is closed.

---

Related Topics