live output from subprocess command
TLDR for Python 3:
import subprocess
import sys
with open("test.log", "wb") as f:
process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), b""):
sys.stdout.buffer.write(c)
f.buffer.write(c)
You have two ways of doing this, either by creating an iterator from the read
or readline
functions and do:
import subprocess
import sys
# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
# replace "" with b'' for Python 3
for c in iter(lambda: process.stdout.read(1), ""):
sys.stdout.write(c)
f.write(c)
or
import subprocess
import sys
# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
# replace "" with b"" for Python 3
for line in iter(process.stdout.readline, ""):
sys.stdout.write(line)
f.write(line)
Or you can create a reader
and a writer
file. Pass the writer
to the Popen
and read from the reader
import io
import time
import subprocess
import sys
filename = "test.log"
with io.open(filename, "wb") as writer, io.open(filename, "rb", 1) as reader:
process = subprocess.Popen(command, stdout=writer)
while process.poll() is None:
sys.stdout.write(reader.read())
time.sleep(0.5)
# Read the remaining
sys.stdout.write(reader.read())
This way you will have the data written in the test.log
as well as on the standard output.
The only advantage of the file approach is that your code doesn't block. So you can do whatever you want in the meantime and read whenever you want from the reader
in a non-blocking way. When you use PIPE
, read
and readline
functions will block until either one character is written to the pipe or a line is written to the pipe respectively.
How to get live output with subprocess in Python
There are three layers of buffering here, and you need to limit all three of them to guarantee you get live data:
Use the
stdbuf
command (on Linux) to wrap thesubprocess
execution (e.g. run['stdbuf', '-oL'] + cmd
instead of justcmd
), or (if you have the ability to do so) alter the program itself to either explicitly change the buffering onstdout
(e.g. usingsetvbuf
for C/C++ code to switchstdout
globally to line-buffered mode, rather than the default block buffering it uses when outputting to a non-tty) or to insert flush statements after critical output (e.g.fflush(stdout);
for C/C++,fileobj.flush()
for Python, etc.) the buffering of the program to line-oriented mode (or add fflushs); without that, everything is stuck in user-mode buffers of the sub-process.Add
bufsize=0
to thePopen
arguments (probably not needed since you don't send anything to stdin, but harmless) so it unbuffers all piped handles. If thePopen
is intext=True
mode, switch tobufsize=1
(which is line-buffered, rather than unbuffered).Add
flush=True
to theprint
arguments (if you're connected to a terminal, the line-buffering will flush it for you, so it's only if stdout is piped to a file that this will matter), or explicitly callsys.stdout.flush()
.
Between the three of these, you should be able to guarantee no data is stuck waiting in user-mode buffers; if at least one line has been output by the sub-process, it will reach you immediately, and any output triggered by it will also appear immediately. Item #1 is the hardest in most cases (when you can't use stdbuf
, or the process reconfigures its own buffering internally and undoes the effect of stdbuf
, and you can't modify the process executable to fix it); you have complete control over #2 and #3, but #1 may be outside your control.
Live output status from subprocess command Python
From this answer:
The difference between
check_output
andPopen
is that, whilepopen
is a non-blocking function (meaning you can continue the execution of the program without waiting the call to finish),check_output
is blocking.
Meaning if you are using subprocess.check_output()
, you cannot have a live output.
Try switching to Popen()
.
Live-output / stream from Python subprocess
Had some problems referencing the selected answer for streaming output from a test runner. The following worked better for me:
import subprocess
from time import sleep
def stream_process(process):
go = process.poll() is None
for line in process.stdout:
print(line)
return go
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while stream_process(process):
sleep(0.1)
Getting realtime output using subprocess
I tried this, and for some reason while the code
for line in p.stdout:
...
buffers aggressively, the variant
while True:
line = p.stdout.readline()
if not line: break
...
does not. Apparently this is a known bug: http://bugs.python.org/issue3907 (The issue is now "Closed" as of Aug 29, 2018)
Live output status from subprocess command Error:I/o operation on closed file Python
You get this error due to the stdout
file descriptor has already been closed when you want to iterate on it. I have written a working version. This implementation can provide the output of the called command in real-time.
Code:
import sys
import subprocess
cmd = 'netstat -nlpt | grep "java" | grep -v tcp6'
result1 = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, universal_newlines=True
)
while True:
out = result1.stdout.read(1)
if out == "" and result1.poll() is not None:
break
if out != "":
sys.stdout.write(out)
sys.stdout.flush()
Output:
>>> python3 test.py
tcp 0 0 127.0.0.1:6943 0.0.0.0:* LISTEN 239519/java
tcp 0 0 127.0.0.1:63343 0.0.0.0:* LISTEN 239519/java
Related Topics
Extracting Just Month and Year Separately from Pandas Datetime Column
Checking If a String Can Be Converted to Float in Python
How to Run Functions in Parallel
Runtimeerror on Windows Trying Python Multiprocessing
Why Does Adding a Trailing Comma After a Variable Name Make It a Tuple
Python Module for Converting PDF to Text
Using an Numpy Array as Indices of the 2Nd Dim of Another Array
Why Is the Id of a Python Class Not Unique When Called Quickly
What Does the Slash Mean in Help() Output
Animated Sprite from Few Images
How to Find All the Subclasses of a Class Given Its Name