Subprocess Readline Hangs Waiting for Eof

subprocess readline hangs waiting for EOF

As @Ron Reiter pointed out, you can't use readline() because cout doesn't print newlines implicitly -- you either need std::endl or "\n" here.

For an interactive use, when you can't change the child program, pexpect module provides several convenience methods (and in general it solves for free: input/output directly from/to terminal (outside of stdin/stdout) and block-buffering issues):

#!/usr/bin/env python
import sys

if sys.version_info[:1] < (3,):
    from pexpect import spawn, EOF # $ pip install pexpect
else:
    from pexpect import spawnu as spawn, EOF # Python 3

child = spawn("./randomNumber") # run command
child.delaybeforesend = 0 
child.logfile_read = sys.stdout # print child output to stdout for debugging
child.expect("enter a number: ") # read the first prompt
lo, hi = 0, 100
while lo <= hi:
    mid = (lo + hi) // 2
    child.sendline(str(mid)) # send number
    index = child.expect([": ", EOF]) # read prompt
    if index == 0: # got prompt
        prompt = child.before
        if "too high" in prompt:
            hi = mid - 1 # guess > num
        elif "too low" in prompt:
            lo = mid + 1 # guess < num
    elif index == 1: # EOF
        assert "Congratulations" in child.before
        child.close()
        break
else:
    print('not found')
    child.terminate()
sys.exit(-child.signalstatus if child.signalstatus else child.exitstatus)

It works but it is a binary search therefore (traditionally) there could be bugs.

Here's a similar code that uses subprocess module for comparison:

#!/usr/bin/env python
from __future__ import print_function
import sys
from subprocess import Popen, PIPE

p = Popen("./randomNumber", stdin=PIPE, stdout=PIPE,
          bufsize=1, # line-buffering
          universal_newlines=True) # enable text mode
p.stdout.readline() # discard welcome message: "This program gener...

readchar = lambda: p.stdout.read(1)
def read_until(char):
    buf = []
    for c in iter(readchar, char):
        if not c: # EOF
            break
        buf.append(c)
    else: # no EOF
        buf.append(char)
    return ''.join(buf).strip()

prompt = read_until(':') # read 1st prompt
lo, hi = 0, 100
while lo <= hi:
    mid = (lo + hi) // 2
    print(prompt, mid)
    print(mid, file=p.stdin) # send number
    prompt = read_until(':') # read prompt
    if "Congratulations" in prompt:
        print(prompt)
        print(mid)
        break # found
    elif "too high" in prompt:
        hi = mid - 1 # guess > num
    elif "too low" in prompt:
        lo = mid + 1 # guess < num
else:
    print('not found')
    p.kill()
for pipe in [p.stdin, p.stdout]:
    try:
        pipe.close()
    except OSError:
        pass
sys.exit(p.wait())

Python Subprocess readline() hangs; can't use normal options

I finally got a working solution; the key piece of information I was missing was thread.daemon = True, which @augurar pointed out in their answer.

Setting thread.daemon = True allows the thread to be terminated when the main process terminates; therefore unblocking my use of a sub-thread to monitor readline().

Here is a sample implementation of my solution; I used a Queue() object to pass strings to the main process, and I implemented a 3 second timer for cases like the original problem I was trying to solve where the subprocess has finished and terminated, but the readline() is hung for some reason.

This also helps avoid a race condition between which thing finishes first.

This works for both Python 2 and 3.

import sys
import threading
import subprocess
from datetime import datetime

try:
    import queue
except:
    import Queue as queue # Python 2 compatibility

def _monitor_readline(process, q):
    while True:
        bail = True
        if process.poll() is None:
            bail = False
        out = ""
        if sys.version_info[0] >= 3:
            out = process.stdout.readline().decode('utf-8')
        else:
            out = process.stdout.readline()
        q.put(out)
        if q.empty() and bail:
            break

def bash(cmd):
    # Kick off the command
    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)

    # Create the queue instance
    q = queue.Queue()
    # Kick off the monitoring thread
    thread = threading.Thread(target=_monitor_readline, args=(process, q))
    thread.daemon = True
    thread.start()
    start = datetime.now()
    while True:
        bail = True
        if process.poll() is None:
            bail = False
            # Re-set the thread timer
            start = datetime.now()
        out = ""
        while not q.empty():
            out += q.get()
        if out:
            print(out)

        # In the case where the thread is still alive and reading, and
        # the process has exited and finished, give it up to 3 seconds
        # to finish reading
        if bail and thread.is_alive() and (datetime.now() - start).total_seconds() < 3:
            bail = False
        if bail:
            break

# To demonstrate output in realtime, sleep is called in between these echos
bash("echo lol;sleep 2;echo bbq")

Python subprocess readlines() hangs

I assume you use pty due to reasons outlined in Q: Why not just use a pipe (popen())? (all other answers so far ignore your "NOTE: I don't want to print out everything at once").

pty is Linux only as said in the docs:

Because pseudo-terminal handling is highly platform dependent, there
is code to do it only for Linux. (The Linux code is supposed to work
on other platforms, but hasn’t been tested yet.)

It is unclear how well it works on other OSes.

You could try pexpect:

import sys
import pexpect

pexpect.run("ruby ruby_sleep.rb", logfile=sys.stdout)

Or stdbuf to enable line-buffering in non-interactive mode:

from subprocess import Popen, PIPE, STDOUT

proc = Popen(['stdbuf', '-oL', 'ruby', 'ruby_sleep.rb'],
             bufsize=1, stdout=PIPE, stderr=STDOUT, close_fds=True)
for line in iter(proc.stdout.readline, b''):
    print line,
proc.stdout.close()
proc.wait()

Or using pty from stdlib based on @Antti Haapala's answer:

#!/usr/bin/env python
import errno
import os
import pty
from subprocess import Popen, STDOUT

master_fd, slave_fd = pty.openpty()  # provide tty to enable
                                     # line-buffering on ruby's side
proc = Popen(['ruby', 'ruby_sleep.rb'],
             stdin=slave_fd, stdout=slave_fd, stderr=STDOUT, close_fds=True)
os.close(slave_fd)
try:
    while 1:
        try:
            data = os.read(master_fd, 512)
        except OSError as e:
            if e.errno != errno.EIO:
                raise
            break # EIO means EOF on some systems
        else:
            if not data: # EOF
                break
            print('got ' + repr(data))
finally:
    os.close(master_fd)
    if proc.poll() is None:
        proc.kill()
    proc.wait()
print("This is reached!")

All three code examples print 'hello' immediately (as soon as the first EOL is seen).

---

^{leave the old more complicated code example here because it may be referenced and discussed in other posts on SO}

Or using pty based on @Antti Haapala's answer:

import os
import pty
import select
from subprocess import Popen, STDOUT

master_fd, slave_fd = pty.openpty()  # provide tty to enable
                                     # line-buffering on ruby's side
proc = Popen(['ruby', 'ruby_sleep.rb'],
             stdout=slave_fd, stderr=STDOUT, close_fds=True)
timeout = .04 # seconds
while 1:
    ready, _, _ = select.select([master_fd], [], [], timeout)
    if ready:
        data = os.read(master_fd, 512)
        if not data:
            break
        print("got " + repr(data))
    elif proc.poll() is not None: # select timeout
        assert not select.select([master_fd], [], [], 0)[0] # detect race condition
        break # proc exited
os.close(slave_fd) # can't do it sooner: it leads to errno.EIO error
os.close(master_fd)
proc.wait()

print("This is reached!")

Python C program subprocess hangs at for line in iter

It is a block buffering issue.

What follows is an extended for your case version of my answer to Python: read streaming input from subprocess.communicate() question.

Fix stdout buffer in C program directly

stdio-based programs as a rule are line buffered if they are running interactively in a terminal and block buffered when their stdout is redirected to a pipe. In the latter case, you won't see new lines until the buffer overflows or flushed.

To avoid calling fflush() after each printf() call, you could force line buffered output by calling in a C program at the very beginning:

setvbuf(stdout, (char *) NULL, _IOLBF, 0); /* make line buffered stdout */

As soon as a newline is printed the buffer is flushed in this case.

Or fix it without modifying the source of C program

There is stdbuf utility that allows you to change buffering type without modifying the source code e.g.:

from subprocess import Popen, PIPE

process = Popen(["stdbuf", "-oL", "./main"], stdout=PIPE, bufsize=1)
for line in iter(process.stdout.readline, b''):
    print line,
process.communicate() # close process' stream, wait for it to exit

There are also other utilities available, see Turn off buffering in pipe.

Or use pseudo-TTY

To trick the subprocess into thinking that it is running interactively, you could use pexpect module or its analogs, for code examples that use pexpect and pty modules, see Python subprocess readlines() hangs. Here's a variation on the pty example provided there (it should work on Linux):

#!/usr/bin/env python
import os
import pty
import sys
from select import select
from subprocess import Popen, STDOUT

master_fd, slave_fd = pty.openpty()  # provide tty to enable line buffering
process = Popen("./main", stdin=slave_fd, stdout=slave_fd, stderr=STDOUT,
                bufsize=0, close_fds=True)
timeout = .1 # ugly but otherwise `select` blocks on process' exit
# code is similar to _copy() from pty.py
with os.fdopen(master_fd, 'r+b', 0) as master:
    input_fds = [master, sys.stdin]
    while True:
        fds = select(input_fds, [], [], timeout)[0]
        if master in fds: # subprocess' output is ready
            data = os.read(master_fd, 512) # <-- doesn't block, may return less
            if not data: # EOF
                input_fds.remove(master)
            else:
                os.write(sys.stdout.fileno(), data) # copy to our stdout
        if sys.stdin in fds: # got user input
            data = os.read(sys.stdin.fileno(), 512)
            if not data:
                input_fds.remove(sys.stdin)
            else:
                master.write(data) # copy it to subprocess' stdin
        if not fds: # timeout in select()
            if process.poll() is not None: # subprocess ended
                # and no output is buffered <-- timeout + dead subprocess
                assert not select([master], [], [], 0)[0] # race is possible
                os.close(slave_fd) # subproces don't need it anymore
                break
rc = process.wait()
print("subprocess exited with status %d" % rc)

Or use pty via pexpect

pexpect wraps pty handling into higher level interface:

#!/usr/bin/env python
import pexpect

child = pexpect.spawn("/.main")
for line in child:
    print line,
child.close()

Q: Why not just use a pipe (popen())? explains why pseudo-TTY is useful.

Timeout on subprocess readline in Python

Thanks for all the answers!

I found a way to solve my problem by simply using select.poll to peek into standard output.

import select
...
scan_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
poll_obj = select.poll()
poll_obj.register(scan_process.stdout, select.POLLIN)
while(some_criterium and not time_limit):
    poll_result = poll_obj.poll(0)
    if poll_result:
        line = scan_process.stdout.readline()
        some_criterium = do_something(line)
    update(time_limit)

Python Popen().stdout.read() hang

You probably want to use .communicate() rather than .wait() plus .read(). Note the warning about wait() on the subprocess documentation page:

Warning This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

http://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait

---

Related Topics