Read Subprocess Stdout Line by Line

read subprocess stdout line by line

I think the problem is with the statement for line in proc.stdout, which reads the entire input before iterating over it. The solution is to use readline() instead:

#filters output
import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
while True:
  line = proc.stdout.readline()
  if not line:
    break
  #the real code does filtering here
  print "test:", line.rstrip()

Of course you still have to deal with the subprocess' buffering.

Note: according to the documentation the solution with an iterator should be equivalent to using readline(), except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).

failed to read subprocess Output line by line

You see empty output because process has been started in a new process but probably haven't finished yet (subprocess.Popen). When you do everything line by line - you paste commands slower than they get executed.

For a better way - see a solution here

The idea is to wrap in "with" construction

with subprocess.Popen(...) as p:
  pass # your code here

Python read from subprocess stdout and stderr separately while preserving order

Here's a solution based on selectors, but one that preserves order, and streams variable-length characters (even single chars).

The trick is to use read1(), instead of read().

import selectors
import subprocess
import sys

p = subprocess.Popen(
    ["python", "random_out.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

sel = selectors.DefaultSelector()
sel.register(p.stdout, selectors.EVENT_READ)
sel.register(p.stderr, selectors.EVENT_READ)

while True:
    for key, _ in sel.select():
        data = key.fileobj.read1().decode()
        if not data:
            exit()
        if key.fileobj is p.stdout:
            print(data, end="")
        else:
            print(data, end="", file=sys.stderr)

If you want a test program, use this.

import sys
from time import sleep


for i in range(10):
    print(f" x{i} ", file=sys.stderr, end="")
    sleep(0.1)
    print(f" y{i} ", end="")
    sleep(0.1)

how to read subprocess stdout using a different separator from line break

You could read a char at a time checking for delimiters:

p = Popen(["command", "args"], stdout=PIPE, universal_newlines=True)
tmp = ""
delims = {"#", "!"}
for ch in iter(lambda: p.stdout.read(1), ""):
    if ch in delims:
        print(tmp)
        tmp = ""
    else:
        tmp += ch

Not the prettiest solution but if you have multiple delimiters I don't see many other ways to do it.

Python subprocess readlines()?

With subprocess.Popen, use communicate to read and write data:

out, err = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE).communicate() 

Then you can always split the string from the processes' stdout with splitlines().

out = out.splitlines()

Read subprocess stdout while maintaining it in the buffer

You can read stdout line by line, process it and save it to a list or buffer, and have the buffer available later. In this example processing is just print, but you could change that however you want. I also assumed you just want to collect stderr in the background, so created a separate thread.

import subprocess as subp
import threading
import io

def _pipe_read_thread(stream, output):
    output.write(stream.read())
    stream.close()

def proc_runner(cmd):
    stdout_lines = []
    stdout_buf = io.BytesIO()
    stderr_buf = io.BytesIO()
    p = subp.Popen(cmd, stdout=subp.PIPE, stderr=subp.PIPE)
    stderr_t = threading.Thread(target=_pipe_read_thread,
        args=(p.stderr, stderr_buf))
    stderr_t.start()
    for line in p.stdout:
        print(line)
        stdout_buf.write(line)
    returncode = p.wait()
    stderr_t.join()
    stdout_buf. seek(0)
    stderr_buf.seek(0)
    return returncode, stdout_buf, stderr_buf

returncode, stdout, stderr = proc_runner(['ls', '-a'])
print('=============================')
print(stdout.read())
print('=============================')
print(stderr.read())

Constantly print Subprocess output while process is running

You can use iter to process lines as soon as the command outputs them: lines = iter(fd.readline, ""). Here's a full example showing a typical use case (thanks to @jfs for helping out):

from __future__ import print_function # Only Python 2.x
import subprocess

def execute(cmd):
    popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
    for stdout_line in iter(popen.stdout.readline, ""):
        yield stdout_line 
    popen.stdout.close()
    return_code = popen.wait()
    if return_code:
        raise subprocess.CalledProcessError(return_code, cmd)

# Example
for path in execute(["locate", "a"]):
    print(path, end="")

---

Related Topics