What's a good equivalent to subprocess.check_call that returns the contents of stdout?
Python 2.7+
from subprocess import check_output as qx
Python < 2.7
From subprocess.py:
import subprocess
def check_output(*popenargs, **kwargs):
if 'stdout' in kwargs:
raise ValueError('stdout argument not allowed, it will be overridden.')
process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
output, unused_err = process.communicate()
retcode = process.poll()
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
raise subprocess.CalledProcessError(retcode, cmd, output=output)
return output
class CalledProcessError(Exception):
def __init__(self, returncode, cmd, output=None):
self.returncode = returncode
self.cmd = cmd
self.output = output
def __str__(self):
return "Command '%s' returned non-zero exit status %d" % (
self.cmd, self.returncode)
# overwrite CalledProcessError due to `output` keyword might be not available
subprocess.CalledProcessError = CalledProcessError
See also Capturing system command output as a string for another example of possible check_output()
implementation.
Why the subprocess.check_call commands always return 0?
The purpose of subprocess.check_call()
is to either return 0
or raise an exception if the exit status of the called process was not 0:
Run command with arguments. Wait for command to complete. If the return code was zero then return, otherwise raise
CalledProcessError
.
Use subprocess.check_output()
instead if you need to read the output of the other command:
Run command with arguments and return its output as a byte string.
The function was added in Python 2.7; if you are using an earlier version of Python, here is a backport:
from subprocess import Popen, PIPE
from subprocess import CalledProcessError as BaseCalledProcessError
class CalledProcessError(BaseCalledProcessError):
def __init__(self, returncode, cmd, output=None):
super(CalledProcessError, self).__init__(returncode, cmd)
self.output = output
def check_output(*popenargs, **kwargs):
r"""Run command with arguments and return its output as a byte string.
If the exit code was non-zero it raises a CalledProcessError. The
CalledProcessError object will have the return code in the returncode
attribute and output in the output attribute.
The arguments are the same as for the Popen constructor. Example:
>>> check_output(["ls", "-l", "/dev/null"])
'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n'
The stdout argument is not allowed as it is used internally.
To capture standard error in the result, use stderr=STDOUT.
>>> check_output(["/bin/sh", "-c",
... "ls -l non_existent_file ; exit 0"],
... stderr=STDOUT)
'ls: non_existent_file: No such file or directory\n'
"""
if 'stdout' in kwargs:
raise ValueError('stdout argument not allowed, it will be overridden.')
process = Popen(stdout=PIPE, *popenargs, **kwargs)
output, unused_err = process.communicate()
retcode = process.poll()
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
raise CalledProcessError(retcode, cmd, output=output)
return output
How to suppress or capture the output of subprocess.run()?
Here is how to suppress output, in order of decreasing levels of cleanliness. They assume you are on Python 3.
- You can redirect to the special
subprocess.DEVNULL
target.
import subprocess
# To redirect stdout (only):
subprocess.run(['ls', '-l'], stdout=subprocess.DEVNULL)
# to redirect stderr to /dev/null as well:
subprocess.run(['ls', '-l'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
# Alternatively, you can merge stderr and stdout streams and redirect
# the one stream to /dev/null
subprocess.run(['ls', '-l'], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
- If you want a fully manual method, can redirect to
/dev/null
by opening the file handle yourself. Everything else would be identical to method #1.
import os
import subprocess
with open(os.devnull, 'w') as devnull:
subprocess.run(['ls', '-l'], stdout=devnull)
Here is how to capture output (to use later or parse), in order of decreasing levels of cleanliness. They assume you are on Python 3.
NOTE: The below examples use
text=True
.
- This causes the STDOUT and STDERR to be captured as
str
instead ofbytes
.
- Omit
text=True
to getbytes
datatext=True
is Python >= 3.7 only, useuniversal_newlines=True
on Python <= 3.6
universal_newlines=True
is identical totext=True
but more verbose to type but should exist on all Python versions
- If you simply want to capture both STDOUT and STDERR independently, AND you are on Python >= 3.7, use
capture_output=True
.
import subprocess
result = subprocess.run(['ls', '-l'], capture_output=True, text=True)
print(result.stdout)
print(result.stderr)
- You can use
subprocess.PIPE
to capture STDOUT and STDERR independently. This works on any version of Python that supportssubprocess.run
.
import subprocess
result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE, text=True)
print(result.stdout)
# To also capture stderr...
result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(result.stdout)
print(result.stderr)
# To mix stdout and stderr into a single string
result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
print(result.stdout)
Store output of subprocess.Popen call in a string
In Python 2.7 or Python 3
Instead of making a Popen
object directly, you can use the subprocess.check_output()
function to store output of a command in a string:
from subprocess import check_output
out = check_output(["ntpq", "-p"])
In Python 2.4-2.6
Use the communicate
method.
import subprocess
p = subprocess.Popen(["ntpq", "-p"], stdout=subprocess.PIPE)
out, err = p.communicate()
out
is what you want.
Important note about the other answers
Note how I passed in the command. The "ntpq -p"
example brings up another matter. Since Popen
does not invoke the shell, you would use a list of the command and options—["ntpq", "-p"]
.
Python subprocess return parameter
You may receive a result and pass a parameter. Terminology is not correct :)
Process returns a result of invocation. It would be 0 (success) or not zero (error condition).
Subprocess.Popen() is for your needs. Pass input to STDIN and get output from STDOUT. Called process must drop their results to STDOUT. Or use another IPC.
Python Subprocess storing call output to file
use the os
module to check if the directory exists and to create it if it does not potential race condition:
import os
if not os.path.isdir('/var/log/messagelog'):
os.mkdir('/var/log/messagelog')
You can also just try to create it and check for specific error:
import os
try:
os.mkdir(path)
except OSError as e:
if e.errno == 17:
print("dir exists")
else:
raise
You can use the EEXIST
constant if it makes it more readable:
import errno
import os
try:
os.mkdir(path)
except OSError as e:
if errno.EEXIST == e.errno:
print("dir exists")
else:
raise
This will only catch a directory exists error, any other OSError will be raised which may or may not be what you want.
Running shell command and capturing the output
In all officially maintained versions of Python, the simplest approach is to use the subprocess.check_output
function:
>>> subprocess.check_output(['ls', '-l'])
b'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'
check_output
runs a single program that takes only arguments as input.1 It returns the result exactly as printed to stdout
. If you need to write input to stdin
, skip ahead to the run
or Popen
sections. If you want to execute complex shell commands, see the note on shell=True
at the end of this answer.
The check_output
function works in all officially maintained versions of Python. But for more recent versions, a more flexible approach is available.
Modern versions of Python (3.5 or higher): run
If you're using Python 3.5+, and do not need backwards compatibility, the new run
function is recommended by the official documentation for most tasks. It provides a very general, high-level API for the subprocess
module. To capture the output of a program, pass the subprocess.PIPE
flag to the stdout
keyword argument. Then access the stdout
attribute of the returned CompletedProcess
object:
>>> import subprocess
>>> result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE)
>>> result.stdout
b'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'
The return value is a bytes
object, so if you want a proper string, you'll need to decode
it. Assuming the called process returns a UTF-8-encoded string:
>>> result.stdout.decode('utf-8')
'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'
This can all be compressed to a one-liner if desired:
>>> subprocess.run(['ls', '-l'], stdout=subprocess.PIPE).stdout.decode('utf-8')
'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'
If you want to pass input to the process's stdin
, you can pass a bytes
object to the input
keyword argument:
>>> cmd = ['awk', 'length($0) > 5']
>>> ip = 'foo\nfoofoo\n'.encode('utf-8')
>>> result = subprocess.run(cmd, stdout=subprocess.PIPE, input=ip)
>>> result.stdout.decode('utf-8')
'foofoo\n'
You can capture errors by passing stderr=subprocess.PIPE
(capture to result.stderr
) or stderr=subprocess.STDOUT
(capture to result.stdout
along with regular output). If you want run
to throw an exception when the process returns a nonzero exit code, you can pass check=True
. (Or you can check the returncode
attribute of result
above.) When security is not a concern, you can also run more complex shell commands by passing shell=True
as described at the end of this answer.
Later versions of Python streamline the above further. In Python 3.7+, the above one-liner can be spelled like this:
>>> subprocess.run(['ls', '-l'], capture_output=True, text=True).stdout
'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'
Using run
this way adds just a bit of complexity, compared to the old way of doing things. But now you can do almost anything you need to do with the run
function alone.
Older versions of Python (3-3.4): more about check_output
If you are using an older version of Python, or need modest backwards compatibility, you can use the check_output
function as briefly described above. It has been available since Python 2.7.
subprocess.check_output(*popenargs, **kwargs)
It takes takes the same arguments as Popen
(see below), and returns a string containing the program's output. The beginning of this answer has a more detailed usage example. In Python 3.5+, check_output
is equivalent to executing run
with check=True
and stdout=PIPE
, and returning just the stdout
attribute.
You can pass stderr=subprocess.STDOUT
to ensure that error messages are included in the returned output. When security is not a concern, you can also run more complex shell commands by passing shell=True
as described at the end of this answer.
If you need to pipe from stderr
or pass input to the process, check_output
won't be up to the task. See the Popen
examples below in that case.
Complex applications and legacy versions of Python (2.6 and below): Popen
If you need deep backwards compatibility, or if you need more sophisticated functionality than check_output
or run
provide, you'll have to work directly with Popen
objects, which encapsulate the low-level API for subprocesses.
The Popen
constructor accepts either a single command without arguments, or a list containing a command as its first item, followed by any number of arguments, each as a separate item in the list. shlex.split
can help parse strings into appropriately formatted lists. Popen
objects also accept a host of different arguments for process IO management and low-level configuration.
To send input and capture output, communicate
is almost always the preferred method. As in:
output = subprocess.Popen(["mycmd", "myarg"],
stdout=subprocess.PIPE).communicate()[0]
Or
>>> import subprocess
>>> p = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE,
... stderr=subprocess.PIPE)
>>> out, err = p.communicate()
>>> print out
.
..
foo
If you set stdin=PIPE
, communicate
also allows you to pass data to the process via stdin
:
>>> cmd = ['awk', 'length($0) > 5']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
... stderr=subprocess.PIPE,
... stdin=subprocess.PIPE)
>>> out, err = p.communicate('foo\nfoofoo\n')
>>> print out
foofoo
Note Aaron Hall's answer, which indicates that on some systems, you may need to set stdout
, stderr
, and stdin
all to PIPE
(or DEVNULL
) to get communicate
to work at all.
In some rare cases, you may need complex, real-time output capturing. Vartec's answer suggests a way forward, but methods other than communicate
are prone to deadlocks if not used carefully.
As with all the above functions, when security is not a concern, you can run more complex shell commands by passing shell=True
.
Notes
1. Running shell commands: the shell=True
argument
Normally, each call to run
, check_output
, or the Popen
constructor executes a single program. That means no fancy bash-style pipes. If you want to run complex shell commands, you can pass shell=True
, which all three functions support. For example:
>>> subprocess.check_output('cat books/* | wc', shell=True, text=True)
' 1299377 17005208 101299376\n'
However, doing this raises security concerns. If you're doing anything more than light scripting, you might be better off calling each process separately, and passing the output from each as an input to the next, via
run(cmd, [stdout=etc...], input=other_output)
Or
Popen(cmd, [stdout=etc...]).communicate(other_output)
The temptation to directly connect pipes is strong; resist it. Otherwise, you'll likely see deadlocks or have to do hacky things like this.
Using subprocess.Popen for Process with Large Output
You're doing blocking reads to two files; the first needs to complete before the second starts. If the application writes a lot to stderr
, and nothing to stdout
, then your process will sit waiting for data on stdout
that isn't coming, while the program you're running sits there waiting for the stuff it wrote to stderr
to be read (which it never will be--since you're waiting for stdout
).
There are a few ways you can fix this.
The simplest is to not intercept stderr
; leave stderr=None
. Errors will be output to stderr
directly. You can't intercept them and display them as part of your own message. For commandline tools, this is often OK. For other apps, it can be a problem.
Another simple approach is to redirect stderr
to stdout
, so you only have one incoming file: set stderr=STDOUT
. This means you can't distinguish regular output from error output. This may or may not be acceptable, depending on how the application writes output.
The complete and complicated way of handling this is select
(http://docs.python.org/library/select.html). This lets you read in a non-blocking way: you get data whenever data appears on either stdout
or stderr
. I'd only recommend this if it's really necessary. This probably doesn't work in Windows.
Related Topics
Fix Not Load Dynamic Library for Tensorflow Gpu
If Two Variables Point to the Same Object, Why Doesn't Reassigning One Variable Affect the Other
How to Find Unused Functions in Python Code
How to Make a Python Script Executable
Using Django Database Layer Outside of Django
How to Convert a List to a List of Tuples
Understanding .Get() Method in Python
Are Sets Ordered Like Dicts in Python3.6
How to Set "Camera Position" for 3D Plots Using Python/Matplotlib
Importerror: No Module Named <Something>
Integer Overflow in Numpy Arrays
Why Should I Close Files in Python
Flask Importerror: No Module Named Flask
How to Install a Package Inside Virtualenv