Python Multiprocessing: Permission Denied

Python multiprocessing: Permission denied

For POSIX semaphores to work, the users need r/w access to shared memory (/dev/shm).

Check the permissions to /dev/shm. On my laptop (Ubuntu) it looks like this:

$ ls -ld /dev/shm
drwxrwxrwt 2 root root          40 2010-01-05 20:34 shm

To permanently set the correct permissions (even after a reboot), add the following to your /etc/fstab:

none /dev/shm tmpfs rw,nosuid,nodev,noexec 0 0

Haven't tried this, just copied from a forum post.

Using multiprocessing.Manager() with multiple users yields permission denied

Python (at least 2.6) uses a UNIX socket to communicate that appears like so:

/tmp/pymp-eGnU6a/listener-BTHJ0E

We can grab that path and change the permissions on it like so:

#!/usr/bin/env python
import multiprocessing, os, grp, pwd
manager = multiprocessing.Manager()
problematic_list = manager.list()

fullname = manager._address
dirname = os.path.dirname(fullname)

gid = grp.getgrnam('some_group').gr_gid
uid = pwd.getpwnam('root').pw_uid # should always be 0, but you never know

os.chown(dirname, uid, gid)
os.chmod(dirname, 0770)

os.chown(fullname, uid, gid)
os.chmod(fullname, 0770)

os.setgid(gid)
os.setuid(43121) # or whatever your user is
problematic_list.append('anything')

---

WindowsError: Access is denied on calling Process.terminate

While I am still uncertain regarding the precise cause of the problem, I have some additional observations as well as a workaround.

Workaround.

Adding a try-except block in the finally clause.

finally:
    try:
        proc.terminate()
    except WindowsError:
        pass

This also seems to be the solution arrived at in a related (?) issue posted here on GitHub (you may have to scroll down a bit).

Observations.

1. This error is dependent on the size of the object passed to the Process/Queue, but it is not related to the execution of the Process itself. In the OP, the Process completes before the timeout expires.
2. proc.is_alive returns True before and after the execution of proc.terminate() (which then throws the WindowsError). A second or two later, proc.is_alive() returns False and a second call to proc.terminate() succeeds.
3. Forcing the main thread to sleep time.sleep(1) in the finally block also prevents the throwing of the WindowsError. Thanks, @tdelaney's comment in the OP.
4. My best guess is that proc is in the process of freeing memory (?, or something comparable) while being killed by the OS (having completed execution) when the call to proc.terminate() attempts to kill it again.

---

Related Topics