Multiple Inputs and Outputs in Python Subprocess Communicate

Multiple inputs and outputs in python subprocess communicate

This answer should be attributed to @J.F.Sebastian. Thanks for the comments!

The following code got my expected behavior:

import pexpect

analyzer = pexpect.spawn('hfst-lookup analyser-gt-desc.hfstol', encoding='utf-8')
analyzer.expect('> ')

for word in ['слово', 'сработай']:
    print('Trying', word, '...')
    analyzer.sendline(word)
    analyzer.expect('> ')
    print(analyzer.before)

Multiple inputs using subprocess.run in Python 3.7+

How do I pass two inputs (or potentially more) using subprocess.run?

The input function stops reading when it receives a newline character...so you just need to separate your inputs with newlines:

>>> import subprocess
>>> subprocess.run(['python', 'names.py'], input='Joe\nBiden\n', capture_output=True, text=True)
CompletedProcess(args=['python', 'names.py'], returncode=0, stdout='Joe Biden\n', stderr='')

How can I write 2 input to python subprocess

If you create the Popen object with stdin=subprocess.PIPE, it has a stdin attribute that you can write to.

However, the subrocess documentation warns that this can lead to deadlocks:

due to any of the other OS pipe buffers filling up and blocking the child process.

If you write small amounts of data and if the subprocess reads its standard input regularly, it should be fine.

An example:

> python
Python 3.9.13 (main, May 31 2022, 12:56:40) 
[Clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess as sp
>>> proc = sp.Popen(['python', '-i'], stdin=sp.PIPE, text=True)
>>> Python 3.9.13 (main, May 31 2022, 12:56:40) 
[Clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> proc.stdin.write("1+2\n")
4
>>> proc.stdin.flush()
>>> 3
>>> proc.stdin.write("3*9\n")
4
>>> proc.stdin.flush()
>>> 27
>>> proc.stdin.reconfigure(line_buffering=True)
>>> proc.stdin.write("3*12\n")
5
>>> 36
>>> proc.kill()
>>> 

Of note:

• Multiple programs need to be told to be interactive if their standard input is not a terminal. Hence the use of the '-i' flag.
• Use text=True to make your life easier.
• Note that if line_buffering is not activated for a standard input of the subprocess, you have to flush the stream for the process to receive the data.
• Therefore it is probably best to reconfigure the stdin stream of the Popen object and set line_buffering to True.

allowing multiple inputs to python subprocess

You don't need additional processes to pass data to a child process without writing it to disk:

#!/usr/bin/env python
import os
import shutil
import subprocess
import tempfile
import threading
from contextlib import contextmanager    
import pandas as pd

@contextmanager
def named_pipes(count):
    dirname = tempfile.mkdtemp()
    try:
        paths = []
        for i in range(count):
            paths.append(os.path.join(dirname, 'named_pipe' + str(i)))
            os.mkfifo(paths[-1])
        yield paths
    finally:
        shutil.rmtree(dirname)

def write_command_input(df, path):
    df.to_csv(path, header=False,index=False, sep="\t")

dfA = pd.DataFrame([[1,2,3],[3,4,5]], columns=["A","B","C"])
dfB = pd.DataFrame([[5,6,7],[6,7,8]], columns=["A","B","C"])

with named_pipes(2) as paths:
    p = subprocess.Popen(["cat"] + paths, stdout=subprocess.PIPE)
    with p.stdout:
        for df, path in zip([dfA, dfB], paths):
            t = threading.Thread(target=write_command_input, args=[df, path]) 
            t.daemon = True
            t.start()
        result = pd.read_csv(p.stdout, header=None, sep="\t")
p.wait()

cat is used for demonstration. You should use your command instead ("/usr/local/bin/my_command"). I assume that you can't pass the data using standard input and you have to pass input via files. The result is read from subprocess' standard output.

Multiple inputs in a Python Subprocess PIPE (with stdin or communicate)

its simply asking for "do you want to continue?" (yes) and the "admin password"

To answer "yes" if the child process asks "...continue?" and to write a password when prompted:

import pexpect # $ pip install pexpect

output, status = pexpect.run('foo bar',
    events={r'continue\?': 'yes\n', 'password': 'p4$$W__rd\n'},
    withexitstatus=1)

pexpect makes sure that even if the child process writes/reads directly to/from a terminal (outside process' stdout/stdin), you'll see its output and it receives your input.

Python subprocess multiple non blocking communicates

I now switched from using subprocess to using pexpect.
My syntax is now as follows:

child = pexpect.spawn('rosrun ros_pkg ros_node')
command = child.sendline('new command')
output = child.read_nonblocking(10000, timeout=1)
....
logic
....
command = child.sendline('new command')
output = child.read_nonblocking(10000, timeout=1)

Many thanks to novel_yet_trivial on reddit: https://www.reddit.com/r/learnpython/comments/2o2viz/subprocess_popen_multiple_times/

---

Related Topics