Popen Simultaneous Read and Write

Using popen to read and write in C using two executables

  while (fgets(path, sizeof(path), stdout) != NULL) {

you don't want to read from stdout, instead:

  while (fgets(path, sizeof(path), fp) != NULL) {

Simultaneously reading stdin and writing to stdout in python

It looks like your main problem is that you invoke two stream_talker()s and no stream_watcher().

In addition, you probably don't want to busy-wait on your Queue (because that defies the whole point of using a Queue). Your code polls chatter.empty() as fast as it can until the queue has something in it. Use chatter.get() directly instead; it will block until something is available or until it hits your timeout.

Finally, you might save yourself some future confusion if you write to the stream argument in stream_talker(), and not hardcode self.proc.stdin.

python subprocess multiple stdin.write and stdout.read

In general, you should use pexpect for interactive programs (dialog-based interactions).

Your specific issue might be caused by a python version mismatch (you think your code is executed using Python 3 while actually it might be executed using Python 2). The second issue (EOFError) is expected: either catch it in the child script or provide a signal for the child to exit (I use an empty line for that in the code example below).

Here's a Python 3 code that fails loudly on Python 2:

#!/usr/bin/env python3
import sys
from subprocess import Popen, PIPE

with Popen([sys.executable, '-u', 'test.py'], stdin=PIPE, stdout=PIPE,
           universal_newlines=True, bufsize=1) as cat:
    for input_string in ["hello, world!", "and another line", ""]:
        print(input_string, file=cat.stdin, flush=True)
        print(cat.stdout.readline(), end='')

Note:

• sys.exectable is the current python executable (Python 3 in this case)
• universal_newlines=True enables the text mode (otherwise, cat.stdin and cat.stdout work with bytes, not strings)
• -u makes the child's output line-buffered (otherwise, you won't see anything until the child flushes its internal stdout buffer)
• the with-statement closes the pipes and waits for the child process to exit.

And here's the corresponding test.py:

#!/usr/bin/env python3
import time

while True:
    x = input("enter something...")
    if not x: # exit if the input is empty
        break
    print(x)
    time.sleep(1)

Output

enter something...hello, world!
enter something...and another line
enter something...

Note: there is no new line after "enter something..."

It works but it is fragile, read Q: Why not just use a pipe (popen())? and use pexpect instead.

---

If the input is finite and it doesn't depend on the output then you could pass it all at once:

#!/usr/bin/env python3
import sys
from subprocess import check_output

output = check_output([sys.executable, 'test.py'],
                      input="\n".join(["hello, world!", "and another line"]),
                      universal_newlines=True)
print(output, end='')

This version requires that the child handles EOF properly:

#!/usr/bin/env python3
import time

while True:
    try:
        x = input("enter something...")
    except EOFError:
        break # no more input

    print(x)
    time.sleep(1)

The output is the same (as shown above).

How to write to stdout AND to log file simultaneously with Popen?

You can use a pipe to read the data from the program's stdout and write it to all the places you want:

import sys
import subprocess

logfile = open('logfile', 'w')
proc=subprocess.Popen(['cat', 'file'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in proc.stdout:
    sys.stdout.write(line)
    logfile.write(line)
proc.wait()

UPDATE

In python 3, the universal_newlines parameter controls how pipes are used. If False, pipe reads return bytes objects and may need to be decoded (e.g., line.decode('utf-8')) to get a string. If True, python does the decode for you

Changed in version 3.3: When universal_newlines is True, the class uses the encoding locale.getpreferredencoding(False) instead of locale.getpreferredencoding(). See the io.TextIOWrapper class for more information on this change.

How to write and read at the same file using popen in C

You cannot do this with popen, but can build a program using fork, exec and pipe. The last opens two file descriptors, which are related: the parent's connection to a pipe, and the child's connection. To make a two-way connection to a child process, you must use two calls to pipe.

The file-descriptors opened by pipe are not buffered, so you would use read and write to communicate with the child (rather than fgets and fprintf).

For examples and discussion, see

• Does one end of a pipe have both read and write fd?
• Read / Write through a pipe in C
• UNIX pipe() : Example Programs
• pipe(7) - Linux man page
• 6.2.2 Creating Pipes in C

Python read stdin from piped process and from Popen subprocess simultaneously

this code:

for line in proc.stdout.readlines():

is waiting for stdout to close/process to be finished before starting to iterate on the lines because readlines() creates a list of lines. You have to loop on the line generator like this instead:

for line in proc.stdout:

(and in general, it's very rare to have a good reason to use readlines(), even with a "finite" file, when processing line by line, because it uses more memory for nothing)

Python read from subprocess stdout and stderr separately while preserving order

Here's a solution based on selectors, but one that preserves order, and streams variable-length characters (even single chars).

The trick is to use read1(), instead of read().

import selectors
import subprocess
import sys

p = subprocess.Popen(
    ["python", "random_out.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

sel = selectors.DefaultSelector()
sel.register(p.stdout, selectors.EVENT_READ)
sel.register(p.stderr, selectors.EVENT_READ)

while True:
    for key, _ in sel.select():
        data = key.fileobj.read1().decode()
        if not data:
            exit()
        if key.fileobj is p.stdout:
            print(data, end="")
        else:
            print(data, end="", file=sys.stderr)

If you want a test program, use this.

import sys
from time import sleep

for i in range(10):
    print(f" x{i} ", file=sys.stderr, end="")
    sleep(0.1)
    print(f" y{i} ", end="")
    sleep(0.1)

---

Related Topics