Popen.Communicate() Throws Oserror: "[Errno 10] No Child Processes"

Python Subprocess: Too Many Open Files

I guess the problem was due to the fact that I was processing an open file with subprocess:

cmd = "enerCHARMM.pl -par param=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)

Here the cmd variable contain the name of a file that has just been created but not closed. Then the subprocess.Popen calls a system command on that file. After doing this for many times, the program crashed with that error message.

So the message I learned from this is

Close the file you have created, then process it

OSError: [Errno 2] No such file or directory while using python subprocess with command and arguments

Use shell=True if you're passing a string to subprocess.call.

From docs:

If passing a single string, either shell must be True or
else the string must simply name the program to be executed without
specifying any arguments.

subprocess.call(crop, shell=True)

or:

import shlex
subprocess.call(shlex.split(crop))

How to catch exception output from Python subprocess.check_output()?

According to the subprocess.check_output() docs, the exception raised on error has an output attribute that you can use to access the error details:

try:
    subprocess.check_output(...)
except subprocess.CalledProcessError as e:
    print(e.output)

You should then be able to analyse this string and parse the error details with the json module:

if e.output.startswith('error: {'):
    error = json.loads(e.output[7:]) # Skip "error: "
    print(error['code'])
    print(error['message'])

Gunicorn + Subprocesses raises exception [Errno 10]

The error turns out to be related to signal handling of SIGCHLD.

The gunicorn arbiter intercepts SIGCHLD, which breaks subprocess.Popen. The subprocess.Popen module requires that SIGCHLD not be intercepted (at least, this is true for Python 2.6 and earlier).

According to bugs.python.org this bug has been fixed in Python 2.7.

Python: Non-Blocking + Non defunct process

There are a lot of ways to deal with this. The key point is that zombie / "defunct" processes exist so that the parent process can collect their statuses.

1. As the creator of the process, you can announce your intent to ignore the status. The POSIX method is to set the flag SA_NOCLDWAIT (using sigaction). This is a bit of a pain to do in Python; but most Unix-like systems allow you to simply ignore SIGCHLD / SIGCLD (the spelling varies from one Unix-like system to another), which is easy to do in Python:

   import signal

   signal.signal(signal.SIGCHLD, signal.SIG_IGN)

2. Or, if this is not available for some reason or does not work on your system, you can use an old stand-by trick: don't just fork once, fork twice. In the first child, fork a second child; in the second child, use execve (or similar) to run the desired program; and then in the first child, exit (with _exit). In the original parent, use wait or waidpid or whatever the OS provides, and collect the status of the first child.

   The reason this works is that the second child has now become an "orphan" (its parent, the first child, died and was collected by your original process). As an orphan it is handed over to a proxy parent (specifically, to "init") which is always wait-ing and hence collects all the zombies right away.

3. In addition to the double fork, you can make your sub-processes live in their own separate session and/or give up controlling terminal access ("daemonize", in Unix-y terms). (This is a bit messy and OS-dependent; I've coded it before but for some corporate code I don't have access to now.)

4. Finally, you could simply collect those processes periodically. If you're using the subprocess module, simply call the .poll function on each process, whenever it seems convenient. This will return None if the process is still running, and the exit status (having collected it) if it has finished. If some are still running, your main program can exit anyway while they keep running; at that point, they become orphaned, as in method #2 above.

The "ignore SIGCHLD" method is simple and easy but has the drawback of interfering with library routines that create and wait-for sub-processes. There's a work-around in Python 2.7 and later (http://bugs.python.org/issue15756) but it means the library routines can't see any failures in those sub-processes.

[Edit: http://bugs.python.org/issue1731717 is for p.wait(), where p is a process from subprocess.Popen; 15756 is specifically for p.poll(); but in any case if you don't have the fixes, you have to resort to methods 2, 3, or 4.]

---

Related Topics