Intercepting Stdout of a Subprocess While It Is Running

Intercepting stdout of a subprocess while it is running

As Charles already mentioned, the problem is buffering. I ran in to a similar problem when writing some modules for SNMPd, and solved it by replacing stdout with an auto-flushing version.

I used the following code, inspired by some posts on ActiveState:

class FlushFile(object):
    """Write-only flushing wrapper for file-type objects."""
    def __init__(self, f):
        self.f = f
    def write(self, x):
        self.f.write(x)
        self.f.flush()

# Replace stdout with an automatically flushing version
sys.stdout = FlushFile(sys.__stdout__)

Constantly print Subprocess output while process is running

You can use iter to process lines as soon as the command outputs them: lines = iter(fd.readline, ""). Here's a full example showing a typical use case (thanks to @jfs for helping out):

from __future__ import print_function # Only Python 2.x
import subprocess

def execute(cmd):
    popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
    for stdout_line in iter(popen.stdout.readline, ""):
        yield stdout_line 
    popen.stdout.close()
    return_code = popen.wait()
    if return_code:
        raise subprocess.CalledProcessError(return_code, cmd)

# Example
for path in execute(["locate", "a"]):
    print(path, end="")

Real-time intercepting of stdout from another process in Python

Looping over a file unavoidably buffers things in pretty large chunks -- a known issue with all Python 2.* implementations. It works as you intend in Python 3.1, with the final loop being slightly different:

for line in proc.stdout:
    print(">>> " + str(line.rstrip()))

If upgrading to Python 3.1 is impractical (and I know it will often be!), go the other way and write the loop in an old-fashioned manner -- the following version of the loop does work as you intend in Python 2.*:

while True:
    line = proc.stdout.readline()
    if not line:
        break
    print ">>> " + line.rstrip()

Issues intercepting subprocess output in real time

The problem here is that process.stdout.readline() will block until a full line is available. This means the condition line == '' will never be met until the process exits. You have two options around this.

First you can set stdout to non-blocking and manage a buffer yourself. It would look something like this. EDIT: As Terry Jan Reedy pointed out this is a Unix only solution. The second alternative should be preferred.

import fcntl
...

    def startProcess(self):
        self.process = subprocess.Popen(['./subtest.sh'],
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE,
            bufsize=0) # prevent any unnecessary buffering

        # set stdout to non-blocking
        fd = self.process.stdout.fileno()
        fl = fcntl.fcntl(fd, fcntl.F_GETFL)
        fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

        # schedule updatelines
        self.after(100, self.updateLines)

    def updateLines(self):
        # read stdout as much as we can
        line = ''
        while True:
            buff = self.process.stdout.read(1024)
            if buff:
                buff += line.decode()
            else:
                break

        self.console.config(state=tkinter.NORMAL)
        self.console.insert(tkinter.END, line)
        self.console.config(state=tkinter.DISABLED)

        # schedule callback
        if self.process.poll() is None:
            self.after(100, self.updateLines)

The second alternative is to have a separate thread read the lines into a queue. Then have updatelines pop from the queue. It would look something like this

from threading import Thread
from queue import Queue, Empty

def readlines(process, queue):
    while process.poll() is None:
        queue.put(process.stdout.readline())
...

    def startProcess(self):
        self.process = subprocess.Popen(['./subtest.sh'],
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE)

        self.queue = Queue()
        self.thread = Thread(target=readlines, args=(self.process, self.queue))
        self.thread.start()

        self.after(100, self.updateLines)

    def updateLines(self):
        try:
            line = self.queue.get(False) # False for non-blocking, raises Empty if empty
            self.console.config(state=tkinter.NORMAL)
            self.console.insert(tkinter.END, line)
            self.console.config(state=tkinter.DISABLED)
        except Empty:
            pass

        if self.process.poll() is None:
            self.after(100, self.updateLines)

The threading route is probably safer. I'm not positive that setting stdout to non-blocking will work on all platforms.

How to get subprocess output during its running?

I found two solutions, both should works on Linux and Windows and even in Windows Idle:

1. The simplier:

In the parent process print the child's stderr:

#!/usr/bin/python

import logging
import subprocess
import sys

def forever():
    cmd = [sys.executable, 'infinit.py']
    while 1:
        try:
            print 'running new instance of:'
            print ' '.join(cmd)
            popen = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                stderr=subprocess.PIPE, universal_newlines=True)

            for line in iter(popen.stderr.readline, ""):
                print line,

        except Exception as e:
            print 'Something bad happend'
            logging.error(e)

if __name__ == '__main__':
    forever()

For now:

• Parent prints children logging (because logging by default send messages to stderr adn stderr we are printing)
• Parent not print children printings, so:

infinit.py

#!/usr/bin/python

import logging
import sys
import time

logging.basicConfig(level=logging.INFO)

sys.stdout = sys.stderr

i = 0
while 1:
    i += 1
    logging.info('i: {0}'.format(i))
    print i
    time.sleep(0.2)

2. Another approch was here:

Intercepting stdout of a subprocess while it is running

And my code adaptated to the solution

main.py

#!/usr/bin/python

import logging
import subprocess
import sys

def forever():
    CMD = [sys.executable, 'infinit.py']
    while 1:
        try:
            print 'running new instance of:'
            print ' '.join(CMD)

            popen = subprocess.Popen(CMD, stdout=subprocess.PIPE)
            for line in iter(popen.stdout.readline, ""):
                print line,

        except Exception as e:
            print 'Something bad happend'
            logging.error(e)
        finally:
            print

if __name__ == '__main__':
    forever()

inifinit.py

#!/usr/bin/python

import logging
import sys
import time

class FlushFile(object):
    """Write-only flushing wrapper for file-type objects."""
    def __init__(self, f):
        self.f = f
    def write(self, x):
        self.f.write(x)
        self.f.flush()

logging.basicConfig(level=logging.INFO)

sys.stdout = FlushFile(sys.stdout)

i = 0
while 1:
    i += 1
    logging.info('i: {0}'.format(i))
    print i
    time.sleep(0.2)

Check on the stdout of a running subprocess in python

Your second attempt is 90% correct. The only issue is that you are attempting to read all of tail's stdout at the same time once it's finished. However, tail is intended to run (indefinitely?) in the background, so you really want to read stdout from it line-by-line:

from subprocess import Popen, PIPE, STDOUT
p = Popen(["tail", "-f", "/tmp/file"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
for line in p.stdout:
    print(line)

I have removed the shell=True and close_fds=True arguments. The first is unnecessary and potentially dangerous, while the second is just the default.

Remember that file objects are iterable over their lines in Python. The for loop will run until tail dies, but it will process each line as it appears, as opposed to read, which will block until tail dies.

If I create an empty file in /tmp/file, start this program and begin echoing lines into the file using another shell, the program will echo those lines. You should probably replace print with something a bit more useful.

Here is an example of commands I typed after starting the code above:

Command line

$ echo a > /tmp/file
$ echo b > /tmp/file
$ echo c >> /tmp/file

Program Output (From Python in a different shell)

b'a\n'
b'tail: /tmp/file: file truncated\n'
b'b\n'
b'c\n'

In the case that you want your main program be responsive while you respond to the output of tail, start the loop in a separate thread. You should make this thread a daemon so that it does not prevent your program from exiting even if tail is not finished. You can have the thread open the sub-process or you can just pass in the standard output to it. I prefer the latter approach since it gives you more control in the main thread:

def deal_with_stdout():
    for line in p.stdout:
        print(line)

from subprocess import Popen, PIPE, STDOUT
from threading import Thread
p = Popen(["tail", "-f", "/tmp/file"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
t = Thread(target=deal_with_stdout, daemon=True)
t.start()
t.join()

The code here is nearly identical, with the addition of a new thread. I added a join() at the end so the program would behave well as an example (join waits for the thread to die before returning). You probably want to replace that with whatever processing code you would normally be running.

If your thread is complex enough, you may also want to inherit from Thread and override the run method instead of passing in a simple target.

Random behaviour while reading from subprocess stdout in a thread

When you create a thread, it becomes part of the parent process. The parent process is the thread that runs your main function. In your main function, you call stdout_thread.start(), which begins the process of starting a thread and then immediately returns. Aftfer that, there is no more code in your main function, which results in python shutting down the main process. Since your thread is part of the main process, it will be taken down when the main process terminates. Meanwhile, the thread you've started up is still being created.

Here we have what is called a race condition. Your thread is starting while simultaneously the process it belongs to is shutting down. If your thread manages to start up and complete its work before the process terminates, you get your expected result. If the process terminates before the thread has started, you get no output. In the third situation, the process closes its stdout before the thread has finished reading it, resulting in an error.

To fix this, in your main function you should wait for your spawned thread to finish, which could be achieved by calling stdout_thread.join().

Pipe unbuffered stdout from subprocess to websocket

If you write

      for line in p.stdout:

then you (kind of) implicitly say, that you want to wait for a complete line

you had to use read(num_bytes) and not readline()

Below one example to illustrate:

sub.py: (example subprocess)

import sys, time
for v in range(20):
    print(".", end="")
    sys.stdout.flush()
    if v % 4 == 0:
        print()
    if v % 3 != 0:
        time.sleep(0.5)

rdunbuf.py: (example reading stddout unbuffered)

contextlib, time, subprocess

def unbuffered(proc, stream='stdout'):
    stream = getattr(proc, stream)
    with contextlib.closing(stream):
        while True:
            last = stream.read(80) # read up to 80 chars
            # stop when end of stream reached
            if not last:
                if proc.poll() is not None:
                    break
            else:
                yield last

# open subprocess without buffering and without universal_newlines=True
proc = subprocess.Popen(["./sub.py"], stdout=subprocess.PIPE, bufsize=0)

for l in unbuffered(proc):
    print(l)
print("end")

Please note as well, that your code might block if it produces a lot of error messages before producing normal output, as you try first to read all normal output and only then data from stderr.

You should read whataver data your subprocess produces as before any pipeline buffers are blocking independently whether this is stdout or stderr.
You can use select.select() ( https://docs.python.org/3.8/library/select.html#select.select ) in order to decide whether you had to read from stdout, or stderr

---

Related Topics