Python Subprocess Readlines() Hangs

Python subprocess readlines() hangs

I assume you use pty due to reasons outlined in Q: Why not just use a pipe (popen())? (all other answers so far ignore your "NOTE: I don't want to print out everything at once").

pty is Linux only as said in the docs:

Because pseudo-terminal handling is highly platform dependent, there
is code to do it only for Linux. (The Linux code is supposed to work
on other platforms, but hasn’t been tested yet.)

It is unclear how well it works on other OSes.

You could try pexpect:

import sys
import pexpect

pexpect.run("ruby ruby_sleep.rb", logfile=sys.stdout)

Or stdbuf to enable line-buffering in non-interactive mode:

from subprocess import Popen, PIPE, STDOUT

proc = Popen(['stdbuf', '-oL', 'ruby', 'ruby_sleep.rb'],
             bufsize=1, stdout=PIPE, stderr=STDOUT, close_fds=True)
for line in iter(proc.stdout.readline, b''):
    print line,
proc.stdout.close()
proc.wait()

Or using pty from stdlib based on @Antti Haapala's answer:

#!/usr/bin/env python
import errno
import os
import pty
from subprocess import Popen, STDOUT

master_fd, slave_fd = pty.openpty()  # provide tty to enable
                                     # line-buffering on ruby's side
proc = Popen(['ruby', 'ruby_sleep.rb'],
             stdin=slave_fd, stdout=slave_fd, stderr=STDOUT, close_fds=True)
os.close(slave_fd)
try:
    while 1:
        try:
            data = os.read(master_fd, 512)
        except OSError as e:
            if e.errno != errno.EIO:
                raise
            break # EIO means EOF on some systems
        else:
            if not data: # EOF
                break
            print('got ' + repr(data))
finally:
    os.close(master_fd)
    if proc.poll() is None:
        proc.kill()
    proc.wait()
print("This is reached!")

All three code examples print 'hello' immediately (as soon as the first EOL is seen).

---

^{leave the old more complicated code example here because it may be referenced and discussed in other posts on SO}

Or using pty based on @Antti Haapala's answer:

import os
import pty
import select
from subprocess import Popen, STDOUT

master_fd, slave_fd = pty.openpty()  # provide tty to enable
                                     # line-buffering on ruby's side
proc = Popen(['ruby', 'ruby_sleep.rb'],
             stdout=slave_fd, stderr=STDOUT, close_fds=True)
timeout = .04 # seconds
while 1:
    ready, _, _ = select.select([master_fd], [], [], timeout)
    if ready:
        data = os.read(master_fd, 512)
        if not data:
            break
        print("got " + repr(data))
    elif proc.poll() is not None: # select timeout
        assert not select.select([master_fd], [], [], 0)[0] # detect race condition
        break # proc exited
os.close(slave_fd) # can't do it sooner: it leads to errno.EIO error
os.close(master_fd)
proc.wait()

print("This is reached!")

Python Subprocess readline() hangs; can't use normal options

I finally got a working solution; the key piece of information I was missing was thread.daemon = True, which @augurar pointed out in their answer.

Setting thread.daemon = True allows the thread to be terminated when the main process terminates; therefore unblocking my use of a sub-thread to monitor readline().

Here is a sample implementation of my solution; I used a Queue() object to pass strings to the main process, and I implemented a 3 second timer for cases like the original problem I was trying to solve where the subprocess has finished and terminated, but the readline() is hung for some reason.

This also helps avoid a race condition between which thing finishes first.

This works for both Python 2 and 3.

import sys
import threading
import subprocess
from datetime import datetime

try:
    import queue
except:
    import Queue as queue # Python 2 compatibility


def _monitor_readline(process, q):
    while True:
        bail = True
        if process.poll() is None:
            bail = False
        out = ""
        if sys.version_info[0] >= 3:
            out = process.stdout.readline().decode('utf-8')
        else:
            out = process.stdout.readline()
        q.put(out)
        if q.empty() and bail:
            break

def bash(cmd):
    # Kick off the command
    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)

    # Create the queue instance
    q = queue.Queue()
    # Kick off the monitoring thread
    thread = threading.Thread(target=_monitor_readline, args=(process, q))
    thread.daemon = True
    thread.start()
    start = datetime.now()
    while True:
        bail = True
        if process.poll() is None:
            bail = False
            # Re-set the thread timer
            start = datetime.now()
        out = ""
        while not q.empty():
            out += q.get()
        if out:
            print(out)

        # In the case where the thread is still alive and reading, and
        # the process has exited and finished, give it up to 3 seconds
        # to finish reading
        if bail and thread.is_alive() and (datetime.now() - start).total_seconds() < 3:
            bail = False
        if bail:
            break

# To demonstrate output in realtime, sleep is called in between these echos
bash("echo lol;sleep 2;echo bbq")

Python Subprocess readline hangs() after reading all input

readline hangs because your TCP connection is still open and readline expects more data to come in. You must close the connection from server side to notify readline that there is nothing more to read. Usually it is done by closing socket on client side notifying the server that there will not be any more requests to it. When server finishes processing all your commands it closes socket too. And this is the signal for you that you have received all the data that server sent to you.

Or, alternatively, if you don't want to close the connection, you must invent delimiters which will mark end of response. So the client will stop calling readline when such delimiter is read.

subprocess stdout readline hang for special command call

I find the answer for myself: it looks the output of this binary is fully buffered. So change the output to line buffered with stdbuf made me fix this issue:

NAME

stdbuf - Run COMMAND, with modified buffering operations for its standard streams.

-o, --output=MODE
    adjust standard output stream buffering

If MODE is 'L' the corresponding stream will be line buffered.

test.py:

import subprocess
import os
import signal

demo_full_path = "stdbuf -oL ./a2dp_sink_demo"
proc = subprocess.Popen(
    demo_full_path,
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    shell=True,
    preexec_fn=os.setsid,
)
while proc.poll() is None:
    line = proc.stdout.readline().decode()
    print("output is: %s " % line)
    if line.find("Starting BTstack") >= 0:
        break

os.killpg(os.getpgid(proc.pid), signal.SIGTERM)

Execution:

root@cdd1898ee06a:/tmp# python3 test.py
output is: Packet Log: /tmp/hci_dump.pklg

output is: No audio playback.
 tcsetattr(): Inappropriate ioctl for device
tcsetattr ICANON: Inappropriate ioctl for device

output is: Audio will be stored to 'av2dp_sink_demo.wav' file.

output is: Starting BTstack ...

---

Related Topics