Running shell command and capturing the output
In all officially maintained versions of Python, the simplest approach is to use the subprocess.check_output
function:
>>> subprocess.check_output(['ls', '-l'])
b'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'
check_output
runs a single program that takes only arguments as input.1 It returns the result exactly as printed to stdout
. If you need to write input to stdin
, skip ahead to the run
or Popen
sections. If you want to execute complex shell commands, see the note on shell=True
at the end of this answer.
The check_output
function works in all officially maintained versions of Python. But for more recent versions, a more flexible approach is available.
Modern versions of Python (3.5 or higher): run
If you're using Python 3.5+, and do not need backwards compatibility, the new run
function is recommended by the official documentation for most tasks. It provides a very general, high-level API for the subprocess
module. To capture the output of a program, pass the subprocess.PIPE
flag to the stdout
keyword argument. Then access the stdout
attribute of the returned CompletedProcess
object:
>>> import subprocess
>>> result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE)
>>> result.stdout
b'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'
The return value is a bytes
object, so if you want a proper string, you'll need to decode
it. Assuming the called process returns a UTF-8-encoded string:
>>> result.stdout.decode('utf-8')
'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'
This can all be compressed to a one-liner if desired:
>>> subprocess.run(['ls', '-l'], stdout=subprocess.PIPE).stdout.decode('utf-8')
'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'
If you want to pass input to the process's stdin
, you can pass a bytes
object to the input
keyword argument:
>>> cmd = ['awk', 'length($0) > 5']
>>> ip = 'foo\nfoofoo\n'.encode('utf-8')
>>> result = subprocess.run(cmd, stdout=subprocess.PIPE, input=ip)
>>> result.stdout.decode('utf-8')
'foofoo\n'
You can capture errors by passing stderr=subprocess.PIPE
(capture to result.stderr
) or stderr=subprocess.STDOUT
(capture to result.stdout
along with regular output). If you want run
to throw an exception when the process returns a nonzero exit code, you can pass check=True
. (Or you can check the returncode
attribute of result
above.) When security is not a concern, you can also run more complex shell commands by passing shell=True
as described at the end of this answer.
Later versions of Python streamline the above further. In Python 3.7+, the above one-liner can be spelled like this:
>>> subprocess.run(['ls', '-l'], capture_output=True, text=True).stdout
'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'
Using run
this way adds just a bit of complexity, compared to the old way of doing things. But now you can do almost anything you need to do with the run
function alone.
Older versions of Python (3-3.4): more about check_output
If you are using an older version of Python, or need modest backwards compatibility, you can use the check_output
function as briefly described above. It has been available since Python 2.7.
subprocess.check_output(*popenargs, **kwargs)
It takes takes the same arguments as Popen
(see below), and returns a string containing the program's output. The beginning of this answer has a more detailed usage example. In Python 3.5+, check_output
is equivalent to executing run
with check=True
and stdout=PIPE
, and returning just the stdout
attribute.
You can pass stderr=subprocess.STDOUT
to ensure that error messages are included in the returned output. When security is not a concern, you can also run more complex shell commands by passing shell=True
as described at the end of this answer.
If you need to pipe from stderr
or pass input to the process, check_output
won't be up to the task. See the Popen
examples below in that case.
Complex applications and legacy versions of Python (2.6 and below): Popen
If you need deep backwards compatibility, or if you need more sophisticated functionality than check_output
or run
provide, you'll have to work directly with Popen
objects, which encapsulate the low-level API for subprocesses.
The Popen
constructor accepts either a single command without arguments, or a list containing a command as its first item, followed by any number of arguments, each as a separate item in the list. shlex.split
can help parse strings into appropriately formatted lists. Popen
objects also accept a host of different arguments for process IO management and low-level configuration.
To send input and capture output, communicate
is almost always the preferred method. As in:
output = subprocess.Popen(["mycmd", "myarg"],
stdout=subprocess.PIPE).communicate()[0]
Or
>>> import subprocess
>>> p = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE,
... stderr=subprocess.PIPE)
>>> out, err = p.communicate()
>>> print out
.
..
foo
If you set stdin=PIPE
, communicate
also allows you to pass data to the process via stdin
:
>>> cmd = ['awk', 'length($0) > 5']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
... stderr=subprocess.PIPE,
... stdin=subprocess.PIPE)
>>> out, err = p.communicate('foo\nfoofoo\n')
>>> print out
foofoo
Note Aaron Hall's answer, which indicates that on some systems, you may need to set stdout
, stderr
, and stdin
all to PIPE
(or DEVNULL
) to get communicate
to work at all.
In some rare cases, you may need complex, real-time output capturing. Vartec's answer suggests a way forward, but methods other than communicate
are prone to deadlocks if not used carefully.
As with all the above functions, when security is not a concern, you can run more complex shell commands by passing shell=True
.
Notes
1. Running shell commands: the shell=True
argument
Normally, each call to run
, check_output
, or the Popen
constructor executes a single program. That means no fancy bash-style pipes. If you want to run complex shell commands, you can pass shell=True
, which all three functions support. For example:
>>> subprocess.check_output('cat books/* | wc', shell=True, text=True)
' 1299377 17005208 101299376\n'
However, doing this raises security concerns. If you're doing anything more than light scripting, you might be better off calling each process separately, and passing the output from each as an input to the next, via
run(cmd, [stdout=etc...], input=other_output)
Or
Popen(cmd, [stdout=etc...]).communicate(other_output)
The temptation to directly connect pipes is strong; resist it. Otherwise, you'll likely see deadlocks or have to do hacky things like this.
How would I store the shell command output to a variable in python?
By using module subprocess
. It is included in Python's standard library and aims to be the substitute of os.system
. (Note that the parameter capture_output
of subprocess.run
was introduced in Python 3.7)
>>> import subprocess
>>> subprocess.run(['cat', '/etc/hostname'], capture_output=True)
CompletedProcess(args=['cat', '/etc/hostname'], returncode=0, stdout='example.com\n', stderr=b'')
>>> subprocess.run(['cat', '/etc/hostname'], capture_output=True).stdout.decode()
'example.com\n'
In your case, just:
import subprocess
v = subprocess.run(['cat', '/etc/redhat-release'], capture_output=True).stdout.decode()
Update: you can split the shell command easily with shlex.split
provided by the standard library.
>>> import shlex
>>> shlex.split('cat /etc/redhat-release')
['cat', '/etc/redhat-release']
>>> subprocess.run(shlex.split('cat /etc/hostname'), capture_output=True).stdout.decode()
'example.com\n'
Update 2: os.popen
mentioned by @Matthias
However, is is impossible for this function to separate stdout and stderr.
import os
v = os.popen('cat /etc/redhat-release').read()
How to run command in background and also capture the output
Try this Shellcheck-clean code:
#! /bin/bash -p
hostname=$(ssh ... \
| sed -n 's/^.*tunneled with tls termination, //p')
declare -p hostname
- I'm assuming that you don't really want to background the command that generates the output. You just want to run it in a way that allows its output to be captured and filtered. See How do you run multiple programs in parallel from a bash script? for information about how "background" processes are used for parallel processing.
- The
-n
option tosed
means that it doesn't print lines from the input unless explicitly instructed to print. s/^.*tunneled with tls termination, //p
works on input lines that contain anything followed by the stringtunneled with tls termination,
. It deletes everything on the line up to the end of that string and prints the result, which hopefully will be the URL that you want.declare -p varname
is a much more reliable and useful way to show the value of a variable than usingecho
.
shell: capture command output and return status in two different variables
Just capture $?
as you did before. Using the executables true
and false
, we can demonstrate that command substitution does set a proper return code:
$ output=$(true); rc=$?; echo $rc
0
$ output=$(false); rc=$?; echo $rc
1
Multiple command substitutions in one assignment
If more than one command substitution appears in an assignment, the return code of the last command substitution determines the return code of the assignment:
$ output="$(true) $(false)"; rc=$?; echo $rc
1
$ output="$(false) $(true)"; rc=$?; echo $rc
0
Documentation
From the section of man bash
describing variable assignment:
If one of the expansions contained a command substitution, the exit
status of the command is the exit status of the last command
substitution performed. If there were no command substitutions, the
command exits with a status of zero.
Python - Run shell command, get stdout and stderr as a variable, but hide from user?
You can use subprocess.run
function.
from subprocess import run
data = run("ANY COMMAND HERE",capture_output=True,shell=True)
print(data.stdout)
print(data.stderr)
Related Topics
How to Get Pid by Process Name
Get Total Physical Memory in Python
How to Disable CSS in Python Selenium Using Chromeoptions
How to Set Your Pythonpath in an Already-Created Virtualenv
Cross-Platform Subprocess With Hidden Window
Python-Dev Installation Error: Importerror: No Module Named Apt_Pkg
Is There Go Up Line Character? (Opposite of \N)
Take a Screenshot Via a Python Script on Linux
Cannot Kill Python Script With Ctrl-C
Os.Walk Without Hidden Folders
Extracting Text from Ms Word Files in Python
Cqlsh Connection Error: 'Ref() Does Not Take Keyword Arguments'
How to Store the Result of an Executed Shell Command in a Variable in Python
How to Retrieve the Values of Dynamic HTML Content Using Python
Open() in Python Does Not Create a File If It Doesn't Exist
Call Python Script from Bash With Argument
Calling a Python Script from Command Line Without Typing "Python" First