Blocking and Non Blocking subprocess calls
Popen
is nonblocking. call
and check_call
are blocking.
You can make the Popen
instance block by calling its wait
or communicate
method.
If you look in the source code, you'll see call
calls Popen(...).wait()
, which is why it is blocking. check_call
calls call
, which is why it blocks as well.
Strictly speaking, shell=True
is orthogonal to the issue of blocking. However, shell=True
causes Python to exec a shell and then run the command in the shell. If you use a blocking call, the call will return when the shell finishes. Since the shell may spawn a subprocess to run the command, the shell may finish before the spawned subprocess. For example,
import subprocess
import time
proc = subprocess.Popen('ls -lRa /', shell=True)
time.sleep(3)
proc.terminate()
proc.wait()
Here two processes are spawned: Popen spawns one subprocess running the shell. The shell in turn spawns a subprocess running ls
. proc.terminate()
kills the shell, but the subprocess running ls
remains. (That is manifested by copious output, even after the python script has ended. Be prepared to kill the ls
with pkill ls
.)
A non-blocking read on a subprocess.PIPE in Python
fcntl
, select
, asyncproc
won't help in this case.
A reliable way to read a stream without blocking regardless of operating system is to use Queue.get_nowait()
:
import sys
from subprocess import PIPE, Popen
from threading import Thread
try:
from queue import Queue, Empty
except ImportError:
from Queue import Queue, Empty # python 2.x
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
p = Popen(['myprogram.exe'], stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()
# ... do other things here
# read line without blocking
try: line = q.get_nowait() # or q.get(timeout=.1)
except Empty:
print('no output yet')
else: # got line
# ... do something with line
Python subprocess multiple non blocking communicates
I now switched from using subprocess to using pexpect.
My syntax is now as follows:
child = pexpect.spawn('rosrun ros_pkg ros_node')
command = child.sendline('new command')
output = child.read_nonblocking(10000, timeout=1)
....
logic
....
command = child.sendline('new command')
output = child.read_nonblocking(10000, timeout=1)
Many thanks to novel_yet_trivial on reddit: https://www.reddit.com/r/learnpython/comments/2o2viz/subprocess_popen_multiple_times/
Related Topics
Tkinter Gui Layout Using Frames and Grid
Programmatically Saving Image to Django Imagefield
Running Selenium with Headless Chrome Webdriver
Suppress the U'Prefix Indicating Unicode' in Python Strings
Choosing a File in Python with Simple Dialog
Run Python Script Without Windows Console Appearing
Why Does Python Return 0 for Simple Division Calculation
Converting Xml to JSON Using Python
What Does the Percentage Sign Mean in Python
Why am I Getting Attributeerror: Object Has No Attribute
How to Append One String to Another in Python
Find_Element_By_* Commands Are Deprecated in Selenium
How to Add Placeholder to an Entry in Tkinter
How to See If There's an Available and Active Network Connection in Python
Make 2 Functions Run at the Same Time