Constantly print Subprocess output while process is running
You can use iter to process lines as soon as the command outputs them: lines = iter(fd.readline, "")
. Here's a full example showing a typical use case (thanks to @jfs for helping out):
from __future__ import print_function # Only Python 2.x
import subprocess
def execute(cmd):
popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
for stdout_line in iter(popen.stdout.readline, ""):
yield stdout_line
popen.stdout.close()
return_code = popen.wait()
if return_code:
raise subprocess.CalledProcessError(return_code, cmd)
# Example
for path in execute(["locate", "a"]):
print(path, end="")
How to get live output with subprocess in Python
There are three layers of buffering here, and you need to limit all three of them to guarantee you get live data:
Use the
stdbuf
command (on Linux) to wrap thesubprocess
execution (e.g. run['stdbuf', '-oL'] + cmd
instead of justcmd
), or (if you have the ability to do so) alter the program itself to either explicitly change the buffering onstdout
(e.g. usingsetvbuf
for C/C++ code to switchstdout
globally to line-buffered mode, rather than the default block buffering it uses when outputting to a non-tty) or to insert flush statements after critical output (e.g.fflush(stdout);
for C/C++,fileobj.flush()
for Python, etc.) the buffering of the program to line-oriented mode (or add fflushs); without that, everything is stuck in user-mode buffers of the sub-process.Add
bufsize=0
to thePopen
arguments (probably not needed since you don't send anything to stdin, but harmless) so it unbuffers all piped handles. If thePopen
is intext=True
mode, switch tobufsize=1
(which is line-buffered, rather than unbuffered).Add
flush=True
to theprint
arguments (if you're connected to a terminal, the line-buffering will flush it for you, so it's only if stdout is piped to a file that this will matter), or explicitly callsys.stdout.flush()
.
Between the three of these, you should be able to guarantee no data is stuck waiting in user-mode buffers; if at least one line has been output by the sub-process, it will reach you immediately, and any output triggered by it will also appear immediately. Item #1 is the hardest in most cases (when you can't use stdbuf
, or the process reconfigures its own buffering internally and undoes the effect of stdbuf
, and you can't modify the process executable to fix it); you have complete control over #2 and #3, but #1 may be outside your control.
Getting realtime output using subprocess
I tried this, and for some reason while the code
for line in p.stdout:
...
buffers aggressively, the variant
while True:
line = p.stdout.readline()
if not line: break
...
does not. Apparently this is a known bug: http://bugs.python.org/issue3907 (The issue is now "Closed" as of Aug 29, 2018)
Check on the stdout of a running subprocess in python
Your second attempt is 90% correct. The only issue is that you are attempting to read all of tail
's stdout at the same time once it's finished. However, tail
is intended to run (indefinitely?) in the background, so you really want to read stdout from it line-by-line:
from subprocess import Popen, PIPE, STDOUT
p = Popen(["tail", "-f", "/tmp/file"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
for line in p.stdout:
print(line)
I have removed the shell=True
and close_fds=True
arguments. The first is unnecessary and potentially dangerous, while the second is just the default.
Remember that file objects are iterable over their lines in Python. The for
loop will run until tail
dies, but it will process each line as it appears, as opposed to read
, which will block until tail
dies.
If I create an empty file in /tmp/file
, start this program and begin echoing lines into the file using another shell, the program will echo those lines. You should probably replace print
with something a bit more useful.
Here is an example of commands I typed after starting the code above:
Command line
$ echo a > /tmp/file
$ echo b > /tmp/file
$ echo c >> /tmp/file
Program Output (From Python in a different shell)
b'a\n'
b'tail: /tmp/file: file truncated\n'
b'b\n'
b'c\n'
In the case that you want your main program be responsive while you respond to the output of tail
, start the loop in a separate thread. You should make this thread a daemon so that it does not prevent your program from exiting even if tail
is not finished. You can have the thread open the sub-process or you can just pass in the standard output to it. I prefer the latter approach since it gives you more control in the main thread:
def deal_with_stdout():
for line in p.stdout:
print(line)
from subprocess import Popen, PIPE, STDOUT
from threading import Thread
p = Popen(["tail", "-f", "/tmp/file"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
t = Thread(target=deal_with_stdout, daemon=True)
t.start()
t.join()
The code here is nearly identical, with the addition of a new thread. I added a join()
at the end so the program would behave well as an example (join
waits for the thread to die before returning). You probably want to replace that with whatever processing code you would normally be running.
If your thread is complex enough, you may also want to inherit from Thread
and override the run
method instead of passing in a simple target
.
Random behaviour while reading from subprocess stdout in a thread
When you create a thread, it becomes part of the parent process. The parent process is the thread that runs your main function. In your main function, you call stdout_thread.start()
, which begins the process of starting a thread and then immediately returns. Aftfer that, there is no more code in your main function, which results in python shutting down the main process. Since your thread is part of the main process, it will be taken down when the main process terminates. Meanwhile, the thread you've started up is still being created.
Here we have what is called a race condition. Your thread is starting while simultaneously the process it belongs to is shutting down. If your thread manages to start up and complete its work before the process terminates, you get your expected result. If the process terminates before the thread has started, you get no output. In the third situation, the process closes its stdout before the thread has finished reading it, resulting in an error.
To fix this, in your main function you should wait for your spawned thread to finish, which could be achieved by calling stdout_thread.join()
.
Related Topics
How to Install Writable Shared and User Specific Data Files with Setuptools
Error Installing Uwsgi in Virtualenv
Pandas Get Topmost N Records Within Each Group
What Is the Python Equivalent for a Case/Switch Statement
Checking If a String Can Be Converted to Float in Python
Animated Sprite from Few Images
Convert Pandas Column Containing Nans to Dtype 'Int'
Python Extract Pattern Matches
How to Call a Python Function from Node.Js
If Python Is Interpreted, What Are .Pyc Files
Using Python Subprocess.Call() to Launch an Ncurses Process
Make (Install from Source) Python Without Running Tests
How to Make a Discontinuous Axis in Matplotlib
Extracting Just Month and Year Separately from Pandas Datetime Column
What Does the Slash Mean in Help() Output
Error: Pandas Hashtable Keyerror
What's the Correct Way to Convert Bytes to a Hex String in Python 3