Retrieving the Output of Subprocess.Call()

Retrieving the output of subprocess.call()

Output from subprocess.call() should only be redirected to files.

You should use subprocess.Popen() instead. Then you can pass subprocess.PIPE for the stderr, stdout, and/or stdin parameters and read from the pipes by using the communicate() method:

from subprocess import Popen, PIPE

p = Popen(['program', 'arg1'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode

The reasoning is that the file-like object used by subprocess.call() must have a real file descriptor, and thus implement the fileno() method. Just using any file-like object won't do the trick.

See here for more info.

How to capture output of subprocess.call

return_code will have the return code of the process. When a process exists successfully (without error), it returns a code of 0. If it errors, it returns a code of 1 (or something non-zero). If you want the output of the program (that it prints to stdout), this is one way to get it:

p = subprocess.run("vcgencmd measure_temp", shell=True,stdout=subprocess.PIPE)
result = p.stdout.decode()
await bot.say("Your RPI3 temp is currently: {}".format(result))

Store output of subprocess.Popen call in a string

In Python 2.7 or Python 3

Instead of making a Popen object directly, you can use the subprocess.check_output() function to store output of a command in a string:

from subprocess import check_output
out = check_output(["ntpq", "-p"])

In Python 2.4-2.6

Use the communicate method.

import subprocess
p = subprocess.Popen(["ntpq", "-p"], stdout=subprocess.PIPE)
out, err = p.communicate()

out is what you want.

Important note about the other answers

Note how I passed in the command. The "ntpq -p" example brings up another matter. Since Popen does not invoke the shell, you would use a list of the command and options—["ntpq", "-p"].

Python: Obtain output using subprocess.call, not Popen

In the documentation on subprocess.call() the one of the first things I noticed was:

Note:

Do not use stdout=PIPE or stderr=PIPE with this function. As the pipes are not being read in the current process, the child process may block if it generates enough output to a pipe to fill up the OS pipe buffer.

The next thing was the first line in the documentation

Run the command described by args. Wait for command to complete, then return the returncode attribute.

subprocess.call will return the "exit code", an int, generally 0 = success, 1 = something went wrong, etc.

For more infomation on exit codes...http://www.tldp.org/LDP/abs/html/exitcodes.html

Since you want the 'output'from your timer, you might want to revert to

timer_out = subprocess.Popen(command, shell=True, stdout=PIPE, stderr=PIPE, universal_newlines=True)
stout, sterror = timer_out.communicate()

...or something like it.

Using subprocess to get output

shell=True makes subprocess use /bin/sh by default. <<< "here-string" is a bash-ism; pass executable='/bin/bash':

>>> import subprocess
>>> subprocess.call(u'cat <<< "\u0061"', shell=True)
/bin/sh: 1: Syntax error: redirection unexpected
2
>>> subprocess.call(u'cat <<< "\u0061"', shell=True, executable='/bin/bash')
a
0

You should also use raw-string literals to avoid escaping backslashes: "\\u0061" == r"\u0061" != u"\u0061":

>>> subprocess.call(r'cat <<< "\u0061"', shell=True, executable='/bin/bash')
\u0061
0

Though you don't need shell=True here. You could pass the input as a string using process.communicate(input=input_string):

>>> process = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> process.communicate(br"\u0061")
('\\u0061', None)

The result could look like:

#!/usr/bin/env python
import shlex
from subprocess import Popen, PIPE

cmd = shlex.split(r'isql -v -b -d, DSN_NAME "DOMAIN\username" password')
process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, errors = process.communicate(
    b"SELECT column_name, data_type "
    b"FROM database_name.information_schema.columns "
    b"WHERE table_name = 'some_table';")

Parse the output of subprocess.call() from Python

Don't PIPE just call check_output passing a list of args and remove shell=True:

 out = subprocess.check_output(["curl", "-k","--data", etree.tostring(tree)+"@SampleRequest.xml", "-v",  "https://world-service-dev.intra.aexp.com:4414/worldservice/CLIC/CaseManagementService/V1"])

If you get a non-zero exit code you will get a CalledProcessError.

How to suppress or capture the output of subprocess.run()?

Here is how to suppress output, in order of decreasing levels of cleanliness. They assume you are on Python 3.

1. You can redirect to the special subprocess.DEVNULL target.

import subprocess

subprocess.run(['ls', '-l'], stdout=subprocess.DEVNULL)
# The above only redirects stdout...
# this will also redirect stderr to /dev/null as well
subprocess.run(['ls', '-l'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
# Alternatively, you can merge stderr and stdout streams and redirect
# the one stream to /dev/null
subprocess.run(['ls', '-l'], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)

2. If you want a fully manual method, can redirect to /dev/null by opening the file handle yourself. Everything else would be identical to method #1.

import os
import subprocess

with open(os.devnull, 'w') as devnull:
    subprocess.run(['ls', '-l'], stdout=devnull)

---

Here is how to capture output (to use later or parse), in order of decreasing levels of cleanliness. They assume you are on Python 3.

NOTE: The below examples use text=True.

• This causes the STDOUT and STDERR to be captured as str instead of bytes.
  
  • Omit text=True to get bytes data
• text=True is Python >= 3.7 only, use universal_newlines=True on Python <= 3.6
  
  • universal_newlines=True is identical to text=True but more verbose to type but should exist on all Python versions

1. If you simply want to capture both STDOUT and STDERR independently, AND you are on Python >= 3.7, use capture_output=True.

import subprocess

result = subprocess.run(['ls', '-l'], capture_output=True, text=True)
print(result.stdout)
print(result.stderr)

2. You can use subprocess.PIPE to capture STDOUT and STDERR independently. This works on any version of Python that supports subprocess.run.

import subprocess

result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE, text=True)
print(result.stdout)

# To also capture stderr...
result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(result.stdout)
print(result.stderr)

# To mix stdout and stderr into a single string
result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
print(result.stdout)

How to capture the output from subprocess.call to a file?

import subprocess
f = open(r'c:\temp\temp.txt','w')
subprocess.call(['dir', r'c:\temp'], shell=True, stdout=f)
f.close()

Can I have subprocess.call write the output of the call to a string?

No, you can't read the output of subprocess.call() directly into a string.

In order to read the output of a command into a string, you need to use subprocess.Popen(), e.g.:

>>> cmd = subprocess.Popen('ls', stdout=subprocess.PIPE)
>>> cmd_out, cmd_err = cmd.communicate()

cmd_out will have the string with the output of the command.

live output from subprocess command

TLDR for Python 3:

import subprocess
import sys

with open("test.log", "wb") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    for c in iter(lambda: process.stdout.read(1), b""):
        sys.stdout.buffer.write(c)
        f.buffer.write(c)

---

You have two ways of doing this, either by creating an iterator from the read or readline functions and do:

import subprocess
import sys

# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b'' for Python 3
    for c in iter(lambda: process.stdout.read(1), ""):
        sys.stdout.write(c)
        f.write(c)

or

import subprocess
import sys

# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b"" for Python 3
    for line in iter(process.stdout.readline, ""):
        sys.stdout.write(line)
        f.write(line)

Or you can create a reader and a writer file. Pass the writer to the Popen and read from the reader

import io
import time
import subprocess
import sys

filename = "test.log"
with io.open(filename, "wb") as writer, io.open(filename, "rb", 1) as reader:
    process = subprocess.Popen(command, stdout=writer)
    while process.poll() is None:
        sys.stdout.write(reader.read())
        time.sleep(0.5)
    # Read the remaining
    sys.stdout.write(reader.read())

This way you will have the data written in the test.log as well as on the standard output.

The only advantage of the file approach is that your code doesn't block. So you can do whatever you want in the meantime and read whenever you want from the reader in a non-blocking way. When you use PIPE, read and readline functions will block until either one character is written to the pipe or a line is written to the pipe respectively.

---

Related Topics