Store output of subprocess.Popen call in a string
In Python 2.7 or Python 3
Instead of making a Popen
object directly, you can use the subprocess.check_output()
function to store output of a command in a string:
from subprocess import check_output
out = check_output(["ntpq", "-p"])
In Python 2.4-2.6
Use the communicate
method.
import subprocess
p = subprocess.Popen(["ntpq", "-p"], stdout=subprocess.PIPE)
out, err = p.communicate()
out
is what you want.
Important note about the other answers
Note how I passed in the command. The "ntpq -p"
example brings up another matter. Since Popen
does not invoke the shell, you would use a list of the command and options—["ntpq", "-p"]
.
Can I have subprocess.call write the output of the call to a string?
No, you can't read the output of subprocess.call() directly into a string.
In order to read the output of a command into a string, you need to use subprocess.Popen(), e.g.:
>>> cmd = subprocess.Popen('ls', stdout=subprocess.PIPE)
>>> cmd_out, cmd_err = cmd.communicate()
cmd_out will have the string with the output of the command.
Retrieving the output of subprocess.call()
Output from subprocess.call()
should only be redirected to files.
You should use subprocess.Popen()
instead. Then you can pass subprocess.PIPE
for the stderr, stdout, and/or stdin parameters and read from the pipes by using the communicate()
method:
from subprocess import Popen, PIPE
p = Popen(['program', 'arg1'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode
The reasoning is that the file-like object used by subprocess.call()
must have a real file descriptor, and thus implement the fileno()
method. Just using any file-like object won't do the trick.
See here for more info.
Store the output of Subprocess.call in a String
x=subprocess.Popen("grep 'xyz-pqr' textfile_5906.txt",shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
output,err=x.communicate()
This way you can have error and output.
How to save a subprocess output as a list of strings for further analysis?
I think you can do what you want by adding the following line before printing the output:
output = output.decode().splitlines()
Calling decode()
will turn the bytes
into a python string (str
), and the splitlines()
turns that into a list-of-strings.
decode()
is bytes
method that decodes them into a string assuming they've been encoded in utf-8 (by default) — it's not documented in the subprocessing
documentation (that I know of).
Pipe subprocess standard output to a variable
To get the output of ls
, use stdout=subprocess.PIPE
.
>>> proc = subprocess.Popen('ls', stdout=subprocess.PIPE)
>>> output = proc.stdout.read()
>>> print output
bar
baz
foo
The command cdrecord --help
outputs to stderr, so you need to pipe that indstead. You should also break up the command into a list of tokens as I've done below, or the alternative is to pass the shell=True
argument but this fires up a fully-blown shell which can be dangerous if you don't control the contents of the command string.
>>> proc = subprocess.Popen(['cdrecord', '--help'], stderr=subprocess.PIPE)
>>> output = proc.stderr.read()
>>> print output
Usage: wodim [options] track1...trackn
Options:
-version print version information and exit
dev=target SCSI target to use as CD/DVD-Recorder
gracetime=# set the grace time before starting to write to #.
...
If you have a command that outputs to both stdout and stderr and you want to merge them, you can do that by piping stderr to stdout and then catching stdout.
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
As mentioned by Chris Morgan, you should be using proc.communicate()
instead of proc.read()
.
>>> proc = subprocess.Popen(['cdrecord', '--help'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> out, err = proc.communicate()
>>> print 'stdout:', out
stdout:
>>> print 'stderr:', err
stderr:Usage: wodim [options] track1...trackn
Options:
-version print version information and exit
dev=target SCSI target to use as CD/DVD-Recorder
gracetime=# set the grace time before starting to write to #.
...
live output from subprocess command
TLDR for Python 3:
import subprocess
import sys
with open("test.log", "wb") as f:
process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), b""):
sys.stdout.buffer.write(c)
f.buffer.write(c)
You have two ways of doing this, either by creating an iterator from the read
or readline
functions and do:
import subprocess
import sys
# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
# replace "" with b'' for Python 3
for c in iter(lambda: process.stdout.read(1), ""):
sys.stdout.write(c)
f.write(c)
or
import subprocess
import sys
# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
# replace "" with b"" for Python 3
for line in iter(process.stdout.readline, ""):
sys.stdout.write(line)
f.write(line)
Or you can create a reader
and a writer
file. Pass the writer
to the Popen
and read from the reader
import io
import time
import subprocess
import sys
filename = "test.log"
with io.open(filename, "wb") as writer, io.open(filename, "rb", 1) as reader:
process = subprocess.Popen(command, stdout=writer)
while process.poll() is None:
sys.stdout.write(reader.read())
time.sleep(0.5)
# Read the remaining
sys.stdout.write(reader.read())
This way you will have the data written in the test.log
as well as on the standard output.
The only advantage of the file approach is that your code doesn't block. So you can do whatever you want in the meantime and read whenever you want from the reader
in a non-blocking way. When you use PIPE
, read
and readline
functions will block until either one character is written to the pipe or a line is written to the pipe respectively.
Related Topics
How to Write Json Data to a File
Why Is "Except: Pass" a Bad Programming Practice
Why Do Python Classes Inherit Object
How to Convert a .Py to .Exe For Python
Open Web in New Tab Selenium + Python
Select Dataframe Rows Between Two Dates
What's the Difference Between Eval, Exec, and Compile
How to Capture Sigint in Python
Retrieve Links from Web Page Using Python and Beautifulsoup
Understanding the "Is" Operator
How to Define a Two-Dimensional Array
How to Do Fuzzy Match Merge With Python Pandas
How to Safely Create a Nested Directory
How to Create a New Column from the Output of Pandas Groupby().Sum()
How to Use Glob() to Find Files Recursively