System-Wide Mutex in Python on Linux

System-wide mutex in Python on Linux

The "traditional" Unix answer is to use file locks. You can use lockf(3) to lock sections of a file so that other processes can't edit it; a very common abuse is to use this as a mutex between processes. The python equivalent is fcntl.lockf.

Traditionally you write the PID of the locking process into the lock file, so that deadlocks due to processes dying while holding the lock are identifiable and fixable.

This gets you what you want, since your lock is in a global namespace (the filesystem) and accessible to all processes. This approach also has the perk that non-Python programs can participate in your locking. The downside is that you need a place for this lock file to live; also, some filesystems don't actually lock correctly, so there's a risk that it will silently fail to achieve exclusion. You win some, you lose some.

How to create a system-wide file lock?

There is support in *NIX to apply locks to files (or parts of files). It requires the processes to co-operate with each other (advisory locking). Each process should always check if a given file or record is locked before accessing it. If this check is not performed there is no built in protection to avoid possible corruption. So in your case both processes have to use file locking for this to work.

It is good idea, if only documentary, to use LOCK_UN (unlock) explicitly, although locks are released automatically when a process exits or when the file is closed.

System-wide global variable / semaphore / mutex in C++/Linux?

You can use a named semaphore if you can get all the processes to agree on a common name.

A named semaphore is identified by a name of the form
/somename; that is, a null-terminated string of up to
NAME_MAX-4 (i.e., 251) characters consisting of an initial
slash, followed by one or more characters, none of which are
slashes. Two processes can operate on the same named
semaphore by passing the same name to sem_open(3).

How can you create a cross-thread cross-process lock in python?

Synchronizing related processes

If you can change your architecture to fork off your processes from the same parent, multiprocessing.Lock() should be enough. For example, this makes the threads run serially:

lock = multiprocessing.Lock()

def thread_proc(lock):
    with lock:
        for i in xrange(0, 10):
            print "IN THREAD", threading.current_thread()
            time.sleep(1)

threads = [threading.Thread(
    target=functools.partial(thread_proc, lock))
    for i in [1, 2]
]
for thread in threads:
    thread.start()

A potential problem might be, that multiprocessing.Lock is slightly underdocumented. I cannot give you a definite reference that multiprocessing.Lock objects are also suitable as thread lock objects.

That said: On Windows, multiprocessing.Lock is implemented using CreateSemaphore(), hence you get a cross-process, threading-safe lock. On Unix systems you get a POSIX semaphore, which has the same properties.

Portability might also be a problem, because not all *NIX systems have the POSIX semaphore (FreeBSD still has a port option to compile Python without POSIX semaphore support).

See also Is there any reason to use threading.Lock over multiprocessing.Lock? and Martijn Pieters comment and answer on Why python multiprocessing manager produce threading locks?

Synchronizing unrelated processes

However, as stated in your question, you have unrelated processes. In that case, you need a named semaphore and Python does not provide those out of the box (although it actually uses named semaphores behind the scenes).

The posix_ipc library exposes those for you. Is also seems to work on all relevant platforms.

Is it possible to use mutex in multiprocessing case on Linux/UNIX ?

Mutual exclusion locks (mutexes) prevent multiple threads
from simultaneously executing critical sections of code that
access shared data (that is, mutexes are used to serialize
the execution of threads). All mutexes must be global. A
successful call for a mutex lock by way of mutex_lock()
will cause another thread that is also trying to lock the
same mutex to block until the owner thread unlocks it by way
of mutex_unlock(). Threads within the same process or
within other processes can share mutexes.

Mutexes can synchronize threads within the same process or
in other processes. Mutexes can be used to synchronize
threads between processes if the mutexes are allocated in
writable memory and shared among the cooperating processes
(see mmap(2)), and have been initialized for this task.

Initialization

Mutexes are either intra-process or inter-process, depending
upon the argument passed implicitly or explicitly to the
initialization of that mutex. A statically allocated mutex
does not need to be explicitly initialized; by default, a
statically allocated mutex is initialized with all zeros
and its scope is set to be within the calling process.

For inter-process synchronization, a mutex needs to be allo-
cated in memory shared between these processes. Since the
memory for such a mutex must be allocated dynamically, the
mutex needs to be explicitly initialized using mutex_init().

Best way to communicate resource lock between processes

It seems that every solution has some drawbacks - either some mechanism or module is not available on all platforms (i.e. Linux only or Windows only), or you may run into error recovery issues with a file-system based approach (as you have already pointed out in your question).

Here is a list of some possible options:

Use Python's `multiprocessing` module

This allows you to create a lock like this:

lock = multiprocessing.Lock()

and to acquire and release it like this:

lock.acquire() 
# do something
lock.release()

Here is a complete example.

Pro: Straightforward to use; cross-platform; no issues with error recovery.

Con: Since you currently have two separate programs, you will have to rearrange your code to start two processes from the same python module.

Use `fnctl` (Linux)

For Linux/Unix systems, there is fcntl (with fcntl.flock()) available as a python module. This is based on lockfiles.

See also this discussion with some recommendations that I am repeating here:

Write the process ID of the locked process to the file for being able to recognize and fix possible deadlocks.
Put your lock files in a temporary location or a RAM file system.

Con: Not cross-platform, available on Linux/Unix systems only.

Use `posix_ipc` (Linux)

For Linux/Unix systems, there is python_ipc (with a Semaphore class) available as a python module.

Pro: Not file-system based, no issues with error recovery.

Con: Not cross-platform, available on Linux/Unix systems only.

Use `msvcrt` (Windows)

For Windows systems, there is msvcrt (with msvcrt.locking()) available as a python module.

Use a third-party library

You might want to check out the following python libraries:

ilock
portalocker
filelock

sharing Lock between subprocesses in python

so thanks to this
System-wide mutex in Python on Linux
i got from it is what i am asking for is a system wide mutex, and you can achive it by using ilock, and this is my example

file 1

from ilock import ILock

print("start this process first")
lock = ILock("VoidLock")
with lock:
    print("now this process inside, run the other procsses")
    input("enter anything so the other procsses can get inside the lock")

print("the lock is relased")

input()

file 2

from ilock import ILock


lock = ILock("VoidLock")
print("now this process is witting")
with lock:
    print("now this process is inside ")
    input()

input()

System-Wide Mutex in Python on Linux