How to Push a Subprocess.Call() Output to Terminal and File

How do I push a subprocess.call() output to terminal and file?

If ddrescue doesn't change its output if its stdout/stderr are redirected to a pipe then you could use tee utility, to display output on the terminal and to save it to a file:

$ ddrescue input_path output_path ddrescue_logfile |& tee logfile

If it does then you could try to provide a pseudo-tty using script utility:

$ script -c 'ddrescue input_path output_path ddrescue_logfile' -q logfile

If it writes directly to a terminal then you could use screen to capture the output:

$ screen -L -- ddrescue input_path output_path ddrescue_logfile

The output is saved in screenlog.0 file by default.

---

To emulate the tee-based command in Python without calling tee utility:

#!/usr/bin/env python3
import shlex
import sys
from subprocess import Popen, PIPE, STDOUT

command = 'ddrescue input_path output_path ddrescue_logfile'
with Popen(shlex.split(command), stdout=PIPE, stderr=STDOUT, bufsize=1) as p:
    with open('logfile', 'wb') as logfile:
        for line in p.stdout:
            logfile.write(line)
            sys.stdout.buffer.write(line)
            sys.stdout.buffer.flush()

To call the tee-based command in Python using shell=True:

#!/usr/bin/env python
from pipes import quote
from subprocess import call

files = input_path, output_path, ddrescue_logfile
rc = call('ddrescue {} |  tee -a drclog'.format(' '.join(map(quote, files))),
          shell=True)

To emulate the script-based command:

#!/usr/bin/env python3
import os
import shlex
import pty

logfile = open('logfile', 'wb')
def read(fd):
    data = os.read(fd, 1024) # doesn't block, it may return less
    logfile.write(data) # it can block but usually not for long
    return data
command = 'ddrescue input_path output_path ddrescue_logfile'
status = pty.spawn(shlex.split(command), read)
logfile.close()

To call screen command in Python:

#!/usr/bin/env python3
import os
import shlex
from subprocess import check_call

screen_cmd = 'screen -L -- ddrescue input_path output_path ddrescue_logfile'
check_call(shlex.split(screen_cmd))
os.replace('screenlog.0', 'logfile')

How do I pipe a subprocess call to a text file?

If you want to write the output to a file you can use the stdout-argument of subprocess.call.

It takes either

• None (the default, stdout is inherited from the parent (your script))
• subprocess.PIPE (allows you to pipe from one command/process to another)
• a file object or a file descriptor (what you want, to have the output written to a file)

You need to open a file with something like open and pass the object or file descriptor integer to call:

f = open("blah.txt", "w")
subprocess.call(["/home/myuser/run.sh", "/tmp/ad_xml",  "/tmp/video_xml"], stdout=f)

I'm guessing any valid file-like object would work, like a socket (gasp :)), but I've never tried.

As marcog mentions in the comments you might want to redirect stderr as well, you can redirect this to the same location as stdout with stderr=subprocess.STDOUT. Any of the above mentioned values works as well, you can redirect to different places.

Retrieving the output of subprocess.call()

Output from subprocess.call() should only be redirected to files.

You should use subprocess.Popen() instead. Then you can pass subprocess.PIPE for the stderr, stdout, and/or stdin parameters and read from the pipes by using the communicate() method:

from subprocess import Popen, PIPE

p = Popen(['program', 'arg1'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode

The reasoning is that the file-like object used by subprocess.call() must have a real file descriptor, and thus implement the fileno() method. Just using any file-like object won't do the trick.

See here for more info.

How to redirect output with subprocess in Python?

UPDATE: os.system is discouraged, albeit still available in Python 3.

---

Use os.system:

os.system(my_cmd)

If you really want to use subprocess, here's the solution (mostly lifted from the documentation for subprocess):

p = subprocess.Popen(my_cmd, shell=True)
os.waitpid(p.pid, 0)

OTOH, you can avoid system calls entirely:

import shutil

with open('myfile', 'w') as outfile:
    for infile in ('file1', 'file2', 'file3'):
        shutil.copyfileobj(open(infile), outfile)

Append to File Using Subprocess in Python

As discussed in this answer, when using, shell=True, you should pass a string, not a list, as the first argument. Thus the correct code would be:

subprocess.call('echo "foo" >> bar.txt', shell=True)

Demonstration:

>>> import subprocess
>>> subprocess.call('echo "foo" >> /tmp/bar.txt', shell=True)
0
>>> open('/tmp/bar.txt').read()
'foo\n'
>>>

append subprocess.Popen output to file?

You sure can append the output of subprocess.Popen to a file, and I make a daily use of it. Here's how I do it:

log = open('some file.txt', 'a')  # so that data written to it will be appended
c = subprocess.Popen(['dir', '/p'], stdout=log, stderr=log, shell=True)

(of course, this is a dummy example, I'm not using subprocess to list files...)

By the way, other objects behaving like file (with write() method in particular) could replace this log item, so you can buffer the output, and do whatever you want with it (write to file, display, etc) [but this seems not so easy, see my comment below].

Note: what may be misleading, is the fact that subprocess, for some reason I don't understand, will write before what you want to write. So, here's the way to use this:

log = open('some file.txt', 'a')
log.write('some text, as header of the file\n')
log.flush()  # <-- here's something not to forget!
c = subprocess.Popen(['dir', '/p'], stdout=log, stderr=log, shell=True)

So the hint is: do not forget to flush the output!

live output from subprocess command

TLDR for Python 3:

import subprocess
import sys

with open("test.log", "wb") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    for c in iter(lambda: process.stdout.read(1), b""):
        sys.stdout.buffer.write(c)
        f.buffer.write(c)

---

You have two ways of doing this, either by creating an iterator from the read or readline functions and do:

import subprocess
import sys

# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b'' for Python 3
    for c in iter(lambda: process.stdout.read(1), ""):
        sys.stdout.write(c)
        f.write(c)

or

import subprocess
import sys

# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b"" for Python 3
    for line in iter(process.stdout.readline, ""):
        sys.stdout.write(line)
        f.write(line)

Or you can create a reader and a writer file. Pass the writer to the Popen and read from the reader

import io
import time
import subprocess
import sys

filename = "test.log"
with io.open(filename, "wb") as writer, io.open(filename, "rb", 1) as reader:
    process = subprocess.Popen(command, stdout=writer)
    while process.poll() is None:
        sys.stdout.write(reader.read())
        time.sleep(0.5)
    # Read the remaining
    sys.stdout.write(reader.read())

This way you will have the data written in the test.log as well as on the standard output.

The only advantage of the file approach is that your code doesn't block. So you can do whatever you want in the meantime and read whenever you want from the reader in a non-blocking way. When you use PIPE, read and readline functions will block until either one character is written to the pipe or a line is written to the pipe respectively.

---

Related Topics