Unbuffered Stdout in Python (As in Python -U) from Within the Program

Unbuffered read from process using subprocess in Python

The file iterator is doing some internal buffering on its own. Try this:

line = p.stdout.readline()
while line:
    print line
    line = p.stdout.readline()

You also need to make sure the process you are running is actually flushing its output buffers frequently.

Unbuffered subprocess output (last line missing)

readline() has to buffer the text, waiting for a new-line.

You'll always have a race condition - no amount of un-buffered streams will handle the fact that you're handling a line, then checking for exit, then reading a line, so if the subprocess exits while you're handling a line, you won't read anything else. In addition the shell is probably introducing its own buffering.

So you could either:

1. Use communicate() and give up the verbose output while the subprocess is running.

2. Make sure you continue reading after the process has exited, until you get EOF.

I would also suggest altering your code so that you don't have to use shell=True.

Python p = Popen([command], stdout=PIPE) when p.stdout.read(-1) is blocked

The child process ./random is using stdio, which by default does "full buffering" rather than "line buffering" when the process's stdout is not a tty device. Since python runs the process via pipes (which are not tty devices) the program is buffering its output, and not printing it at the time of the scanf call. The buffer is flushed (written to stdout) only (a) when it fills up; (b) when explicitly flushed with an fflush call; or (c) at program exit.

You can modify the child process to do explicit fflush calls, or you can use a pseudo-tty from Python, e.g., with pexpect or sh. (Note that the sh module by default uses a pty for output only but this is usually sufficient.) Or, in cases like this where the program you're running only "looks interactive" (but is actually totally predictable), you can just run it with the known input sequence pre-provided:

fd = Popen(["./random"], stdin=PIPE, stdout=PIPE)
result = fd.communicate("write write write!!!\n")[0]

although then of course you'll get all the stdout input mixed together, and have to separate out the two lines it printed (in this particular case).

---

Related Topics