Python Sharing a Lock Between Processes

Python sharing a lock between processes

You can't pass normal multiprocessing.Lock objects to Pool methods, because they can't be pickled. There are two ways to get around this. One is to create Manager() and pass a Manager.Lock():

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    m = multiprocessing.Manager()
    l = m.Lock()
    func = partial(target, l)
    pool.map(func, iterable)
    pool.close()
    pool.join()

This is a little bit heavyweight, though; using a Manager requires spawning another process to host the Manager server. And all calls to acquire/release the lock have to be sent to that server via IPC.

The other option is to pass the regular multiprocessing.Lock() at Pool creation time, using the initializer kwarg. This will make your lock instance global in all the child workers:

def target(iterable_item):
    for item in items:
        # Do cool stuff
        if (... some condition here ...):
            lock.acquire()
            # Write to stdout or logfile, etc.
            lock.release()
def init(l):
    global lock
    lock = l

def main():
    iterable = [1, 2, 3, 4, 5]
    l = multiprocessing.Lock()
    pool = multiprocessing.Pool(initializer=init, initargs=(l,))
    pool.map(target, iterable)
    pool.close()
    pool.join()

The second solution has the side-effect of no longer requiring partial.

Python : sharing a lock between spawned processes

The last code snippet works, provided the script does not exit prematurely. Joining processes is enough :

import multiprocessing as mp
from time import sleep

class OneAtATime:
    def f(self, l):
        with l:
            sleep(1)
        print("Hello")

if __name__ == "__main__":
    mp.set_start_method('spawn')
    a = OneAtATime()
    b = OneAtATime()
    m = mp.Manager()
    l = m.Lock()
    p1 = mp.Process(target = a.f, args = (l,))
    p2 = mp.Process(target = b.f, args = (l,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

More info on the error it was causing here https://stackoverflow.com/a/25456494/8194503.

Python: Why is the multiprocessing lock shared among processes here?

On Windows (which you said you're using), these kinds of things always reduce to details about how multiprocessing plays with pickle, because all Python data crossing process boundaries on Windows is implemented by pickling on the sending end (and unpickling on the receiving end).

My best advice is to avoid doing things that raise such questions to begin with ;-) For example, the code you showed blows up on Windows under Python 2, and also blows up under Python 3 if you use a multiprocessing.Pool method instead of multiprocessing.Process.

It's not just the lock, simply trying to pickle a bound method (like self.run_job) blows up in Python 2. Think about it. You're crossing a process boundary, and there isn't an object corresponding to self on the receiving end. To what object is self.run_job supposed to be bound on the receiving end?

In Python 3, pickling self.run_job also pickles a copy of the self object. So that's the answer: a SampleClass object corresponding to self is created by magic on the receiving end. Clear as mud. t's entire state is pickled, including t.lock. That's why it "works".

See this for more implementation details:

Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?

In the long run, you'll suffer the fewest mysteries if you stick to things that were obviously intended to work: pass module-global callable objects (neither, e.g., instance methods nor local functions), and explicitly pass multiprocessing data objects (whether an instance of Lock, Queue, manager.list, etc etc).

sharing Lock between subprocesses in python

so thanks to this
System-wide mutex in Python on Linux
i got from it is what i am asking for is a system wide mutex, and you can achive it by using ilock, and this is my example

file 1

from ilock import ILock

print("start this process first")
lock = ILock("VoidLock")
with lock:
    print("now this process inside, run the other procsses")
    input("enter anything so the other procsses can get inside the lock")

print("the lock is relased")

input()

file 2

from ilock import ILock

lock = ILock("VoidLock")
print("now this process is witting")
with lock:
    print("now this process is inside ")
    input()

input()

Best way to communicate resource lock between processes

It seems that every solution has some drawbacks - either some mechanism or module is not available on all platforms (i.e. Linux only or Windows only), or you may run into error recovery issues with a file-system based approach (as you have already pointed out in your question).

Here is a list of some possible options:

Use Python's `multiprocessing` module

This allows you to create a lock like this:

lock = multiprocessing.Lock()

and to acquire and release it like this:

lock.acquire() 
# do something
lock.release()

Here is a complete example.

Pro: Straightforward to use; cross-platform; no issues with error recovery.

Con: Since you currently have two separate programs, you will have to rearrange your code to start two processes from the same python module.

Use `fnctl` (Linux)

For Linux/Unix systems, there is fcntl (with fcntl.flock()) available as a python module. This is based on lockfiles.

See also this discussion with some recommendations that I am repeating here:

Write the process ID of the locked process to the file for being able to recognize and fix possible deadlocks.
Put your lock files in a temporary location or a RAM file system.

Con: Not cross-platform, available on Linux/Unix systems only.

Use `posix_ipc` (Linux)

For Linux/Unix systems, there is python_ipc (with a Semaphore class) available as a python module.

Pro: Not file-system based, no issues with error recovery.

Con: Not cross-platform, available on Linux/Unix systems only.

Use `msvcrt` (Windows)

For Windows systems, there is msvcrt (with msvcrt.locking()) available as a python module.

Use a third-party library

You might want to check out the following python libraries:

ilock
portalocker
filelock

Python: Sharing a time Lock in between spawned Processes so that there is a delay between them

I actually fixed this problem. The main reason my code was not joining was because my code was checking if the queue was empty, waiting for a delay, then attempting to get something from the queue. This meant that towards the end of the program that while the queue had become empty and 2 of the 4 processes successfully finished at the same time, the remaining 2 processes were in a delay. When this delay ended they attempted to get something from the queue but since the queue was empty they just blocked the remainder of the process's code from running which meant that they could never join back up.

I fixed this by also checking if the queue is empty right before the process attempts to get something from the queue. My fixed workerfunction is below:

def mp_worker(self, queue, lock):

    while not queue.empty():

        print(mp.current_process().name)
        lock.acquire()
        # Release the lock after a delay
        timer = Timer(self.DELAY, lock.release)
        timer.start()

        if not queue.empty():
            record = queue.get(False)

            start_time = time()
            print("begin {0} : {1}".format(record, start_time))
            if (record % 2 == 0):
                sleep(3)
            else:
                sleep(6)
            print("end {0} : {1}".format(record, time() - start_time))

    print("{0} closed".format(mp.current_process().name))

Why doesn't multiprocessing Lock acquiring work?

Well, call your worker processes "1" and "2". They both start. 2 prints "not locked", sleeps half a second, and loops around to print "not locked" again. But note that what 2 is printing has nothing do with whether lock is locked. Nothing in the code 2 executes even references lock, let alone synchronizes on lock. After another half second, 2 wakes up to print "not locked" for a third time, and goes to sleep again.

While that's going on, 1 starts, acquires the lock, sleeps for 1.1 seconds, and then prints "hi". It then releases the lock and ends. At the time 1 gets around to printing "hi", 2 has already printed "not locked" three times, and is about 0.1 seconds into its latest half-second sleep.

After "hi" is printed, 2 will continue printing "not locked" about twice per second forever more.

So the code appears to be doing what it was told to do.

What I can't guess, though, is how you expected to see "hi" first and then "not locked". That would require some kind of timing miracle, where 2 didn't start executing at all before 1 had been running for over 1.1 seconds. Not impossible, but extremely unlikely.

Changes

Here's one way to get the output you want, although I'm making many guesses about your intent.

If you don't want 2 to start before 1 ends, then you have to force that. One way is to have 2 begin by acquiring lock at the start of what it does. That also requires guaranteeing that lock is in the acquired state before any worker begins.

So acquire it before map() is called. Then there's no point left to having 1 acquire it at all - 1 can just start at once, and release it when it ends, so that 2 can proceed.

There are few changes to the code, but I'll paste all of it in here for convenience:

import multiprocessing
import time
from threading import Lock

def target(arg):
    if arg == 1:
        time.sleep(1.1)
        print('hi')
        lock.release()
    elif arg == 2:
        lock.acquire()
        print('not locked')
        time.sleep(0.5)

def init(lock_: Lock):
    global lock
    lock = lock_

if __name__ == '__main__':
    lock_ = multiprocessing.Lock()
    lock_.acquire()
    with multiprocessing.Pool(initializer=init, initargs=[lock_], processes=2) as pool:
        pool.map(target, [1, 2])

Python Sharing a Lock Between Processes