How to Write to a Python Subprocess' Stdin

How do I write to a Python subprocess' stdin?

It might be better to use communicate:

from subprocess import Popen, PIPE, STDOUT
p = Popen(['myapp'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
stdout_data = p.communicate(input='data_to_write')[0]

"Better", because of this warning:

Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

How do I pass a string into subprocess.Popen (using the stdin argument)?

Popen.communicate() documentation:

Note that if you want to send data to
the process’s stdin, you need to
create the Popen object with
stdin=PIPE. Similarly, to get anything
other than None in the result tuple,
you need to give stdout=PIPE and/or
stderr=PIPE too.

Replacing os.popen*

    pipe = os.popen(cmd, 'w', bufsize)
    # ==>
    pipe = Popen(cmd, shell=True, bufsize=bufsize, stdin=PIPE).stdin

Warning Use communicate() rather than
stdin.write(), stdout.read() or
stderr.read() to avoid deadlocks due
to any of the other OS pipe buffers
filling up and blocking the child
process.

So your example could be written as follows:

from subprocess import Popen, PIPE, STDOUT

p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)    
grep_stdout = p.communicate(input=b'one\ntwo\nthree\nfour\nfive\nsix\n')[0]
print(grep_stdout.decode())
# -> four
# -> five
# ->

---

On Python 3.5+ (3.6+ for encoding), you could use subprocess.run, to pass input as a string to an external command and get its exit status, and its output as a string back in one call:

#!/usr/bin/env python3
from subprocess import run, PIPE

p = run(['grep', 'f'], stdout=PIPE,
        input='one\ntwo\nthree\nfour\nfive\nsix\n', encoding='ascii')
print(p.returncode)
# -> 0
print(p.stdout)
# -> four
# -> five
# -> 

Python subprocess: Giving stdin, reading stdout, then giving more stdin

you don't need to use process.communicate in your example.

Simply read and write using process.stdin.write and process.stdout.read. Also make sure to send a newline, otherwise read won't return. And when you read from stdin, you also have to handle newlines coming from echo.

Note: process.stdout.read will block until EOF.

# talk_with_example_input.py
import subprocess

process = subprocess.Popen(["./exampleInput.sh"], 
                        stdin = subprocess.PIPE,
                        stdout = subprocess.PIPE)

process.stdin.write("5\n")
stdout = process.stdout.readline()
print(stdout)

if stdout == "25\n":
    process.stdin.write("yes\n")
    print(process.stdout.readline())

$ python2 test.py
25

Okay, moving on...

---

Update

When communicating with an program in that way, you have to pay special attention to what the application is actually writing. Best is to analyze the output in a hex editor:

$ chimera --nogui 2>&1 | hexdump -C

Please note that readline ^[1] only reads to the next newline (\n). In your case you have to call readline at least four times to get that first block of output.

If you just want to read everything up until the subprocess stops printing, you have to read byte by byte and implement a timeout. Sadly, neither read nor readline does provide such a timeout mechanism. This is probably because the underlying read syscall ^[2] (Linux) does not provide one either.

On Linux we can write a single-threaded read_with_timeout() using poll / select. For an example see ^[3].

from select import epoll, EPOLLIN

def read_with_timeout(fd, timeout__s):
    """Reads from fd until there is no new data for at least timeout__s seconds.

    This only works on linux > 2.5.44.
    """
    buf = []
    e = epoll()
    e.register(fd, EPOLLIN)
    while True:
        ret = e.poll(timeout__s)
        if not ret or ret[0][1] is not EPOLLIN:
            break
        buf.append(
            fd.read(1)
        )
    return ''.join(buf)

In case you need a reliable way to read non blocking under Windows and Linux, this answer might be helpful.

---

^[1] from the python 2 docs:

readline(limit=-1)

Read and return one line from the stream. If limit is specified, at most limit bytes will be read.

The line terminator is always b'\n' for binary files; for text files, the newline argument to open() can be used to select the line terminator(s) recognized.

^[2] from man 2 read:

#include <unistd.h>

ssize_t read(int fd, void *buf, size_t count);

^[3] example

$ tree
.
├── prog.py
└── prog.sh

prog.sh

#!/usr/bin/env bash

for i in $(seq 3); do
  echo "${RANDOM}"
  sleep 1
done

sleep 3
echo "${RANDOM}"

prog.py

# talk_with_example_input.py
import subprocess
from select import epoll, EPOLLIN

def read_with_timeout(fd, timeout__s):
    """Reads from f until there is no new data for at least timeout__s seconds.

    This only works on linux > 2.5.44.
    """
    buf = []
    e = epoll()
    e.register(fd, EPOLLIN)
    while True:
        ret = e.poll(timeout__s)
        if not ret or ret[0][1] is not EPOLLIN:
            break
        buf.append(
            fd.read(1)
        )
    return ''.join(buf)

process = subprocess.Popen(
    ["./prog.sh"],
    stdin = subprocess.PIPE,
    stdout = subprocess.PIPE
)

print(read_with_timeout(process.stdout, 1.5))
print('-----')
print(read_with_timeout(process.stdout, 3))

$ python2 prog.py 
6194
14508
11293

-----
10506

python subprocess multiple stdin.write and stdout.read

In general, you should use pexpect for interactive programs (dialog-based interactions).

Your specific issue might be caused by a python version mismatch (you think your code is executed using Python 3 while actually it might be executed using Python 2). The second issue (EOFError) is expected: either catch it in the child script or provide a signal for the child to exit (I use an empty line for that in the code example below).

Here's a Python 3 code that fails loudly on Python 2:

#!/usr/bin/env python3
import sys
from subprocess import Popen, PIPE

with Popen([sys.executable, '-u', 'test.py'], stdin=PIPE, stdout=PIPE,
           universal_newlines=True, bufsize=1) as cat:
    for input_string in ["hello, world!", "and another line", ""]:
        print(input_string, file=cat.stdin, flush=True)
        print(cat.stdout.readline(), end='')

Note:

• sys.exectable is the current python executable (Python 3 in this case)
• universal_newlines=True enables the text mode (otherwise, cat.stdin and cat.stdout work with bytes, not strings)
• -u makes the child's output line-buffered (otherwise, you won't see anything until the child flushes its internal stdout buffer)
• the with-statement closes the pipes and waits for the child process to exit.

And here's the corresponding test.py:

#!/usr/bin/env python3
import time

while True:
    x = input("enter something...")
    if not x: # exit if the input is empty
        break
    print(x)
    time.sleep(1)

Output

enter something...hello, world!
enter something...and another line
enter something...

Note: there is no new line after "enter something..."

It works but it is fragile, read Q: Why not just use a pipe (popen())? and use pexpect instead.

---

If the input is finite and it doesn't depend on the output then you could pass it all at once:

#!/usr/bin/env python3
import sys
from subprocess import check_output

output = check_output([sys.executable, 'test.py'],
                      input="\n".join(["hello, world!", "and another line"]),
                      universal_newlines=True)
print(output, end='')

This version requires that the child handles EOF properly:

#!/usr/bin/env python3
import time

while True:
    try:
        x = input("enter something...")
    except EOFError:
        break # no more input

    print(x)
    time.sleep(1)

The output is the same (as shown above).

Write to a Python subprocess's stdin without communicate()'s blocking behavior

The p.communicate method is a one-shot deal in terms of sending data to the process.

Instead, write directly to p.stdin. If you want to get output, you can read lines from p.stdout. Make sure you pass stdout=subprocess.PIPE to the constructor before attempting to read.

How do I pass a string in to subprocess.run using stdin in Python 3

Simplest possible example, send foo to cat and let it print to the screen.

 import subprocess

 subprocess.run(['cat'],input=b'foo\n')

Notice that you send binary data and the carriage return.

---

Related Topics