Interactive input/output using Python
Two solutions for this issue on Linux:
First one is to use a file to write the output to, and read from it simultaneously:
from subprocess import Popen, PIPE
fw = open("tmpout", "wb")
fr = open("tmpout", "r")
p = Popen("./a.out", stdin = PIPE, stdout = fw, stderr = fw, bufsize = 1)
p.stdin.write("1\n")
out = fr.read()
p.stdin.write("5\n")
out = fr.read()
fw.close()
fr.close()
Second, as J.F. Sebastian offered, is to make p.stdout and p.stderr pipes non-blocking using fnctl module:
import os
import fcntl
from subprocess import Popen, PIPE
def setNonBlocking(fd):
"""
Set the file description of the given file descriptor to non-blocking.
"""
flags = fcntl.fcntl(fd, fcntl.F_GETFL)
flags = flags | os.O_NONBLOCK
fcntl.fcntl(fd, fcntl.F_SETFL, flags)
p = Popen("./a.out", stdin = PIPE, stdout = PIPE, stderr = PIPE, bufsize = 1)
setNonBlocking(p.stdout)
setNonBlocking(p.stderr)
p.stdin.write("1\n")
while True:
try:
out1 = p.stdout.read()
except IOError:
continue
else:
break
out1 = p.stdout.read()
p.stdin.write("5\n")
while True:
try:
out2 = p.stdout.read()
except IOError:
continue
else:
break
Running interactive program from within python
You need to keep interacting with your subprocess - at the moment once you pick the output from your subprocess you're pretty much done as you close its STDOUT stream.
Here is the most rudimentary way to continue user input -> process output cycle:
import subprocess
import sys
import time
if __name__ == "__main__": # a guard from unintended usage
input_buffer = sys.stdin # a buffer to get the user input from
output_buffer = sys.stdout # a buffer to write rasa's output to
proc = subprocess.Popen(["path/to/rasa", "arg1", "arg2", "etc."], # start the process
stdin=subprocess.PIPE, # pipe its STDIN so we can write to it
stdout=output_buffer, # pipe directly to the output_buffer
universal_newlines=True)
while True: # run a main loop
time.sleep(0.5) # give some time for `rasa` to forward its STDOUT
print("Input: ", end="", file=output_buffer, flush=True) # print the input prompt
print(input_buffer.readline(), file=proc.stdin, flush=True) # forward the user input
You can replace input_buffer with a buffer coming from your remote user(s) and output_buffer with a buffer that forwards the data to your user(s) and you'll get essentially what you're looking for - the sub-process will be getting the input directly from the user (input_buffer) and print its output to the user (output_buffer).
If you need to perform other tasks while all this is running in the background, just run everything under the if __name__ == "__main__": guard in a separate thread, and I'd suggest adding a try..except block to pick up KeyboardInterrupt and exit gracefully.
But... soon enough you'll notice that it doesn't exactly work properly all the time - if it takes longer than half a second of wait for rasa to print its STDOUT and enter the wait for STDIN stage, the outputs will start to mix. This problem is considerably more complex than you might expect. The main issue is that STDOUT and STDIN (and STDERR) are separate buffers and you cannot know when a subprocess is actually expecting something on its STDIN. This means that without a clear indication from the subprocess (like you have the \r\n[path]> in Windows CMD prompt on its STDOUT for example) you can only send data to the subprocesses STDIN and hope it will be picked up.
Based on your screenshot, it doesn't really give a distinguishable STDIN request prompt because the first prompt is ... :\n and then it waits for STDIN, but then once the command is sent it lists options without an indication of its end of STDOUT stream (technically making the prompt just ...\n but that would match any line preceding it as well). Maybe you can be clever and read the STDOUT line by line, then on each new line measure how much time has passed since the sub-process wrote to it and once a threshold of inactivity is reached assume that rasa expects input and prompt the user for it. Something like:
import subprocess
import sys
import threading
# we'll be using a separate thread and a timed event to request the user input
def timed_user_input(timer, wait, buffer_in, buffer_out, buffer_target):
while True: # user input loop
timer.wait(wait) # wait for the specified time...
if not timer.is_set(): # if the timer was not stopped/restarted...
print("Input: ", end="", file=buffer_out, flush=True) # print the input prompt
print(buffer_in.readline(), file=buffer_target, flush=True) # forward the input
timer.clear() # reset the 'timer' event
if __name__ == "__main__": # a guard from unintended usage
input_buffer = sys.stdin # a buffer to get the user input from
output_buffer = sys.stdout # a buffer to write rasa's output to
proc = subprocess.Popen(["path/to/rasa", "arg1", "arg2", "etc."], # start the process
stdin=subprocess.PIPE, # pipe its STDIN so we can write to it
stdout=subprocess.PIPE, # pipe its STDIN so we can process it
universal_newlines=True)
# lets build a timer which will fire off if we don't reset it
timer = threading.Event() # a simple Event timer
input_thread = threading.Thread(target=timed_user_input,
args=(timer, # pass the timer
1.0, # prompt after one second
input_buffer, output_buffer, proc.stdin))
input_thread.daemon = True # no need to keep the input thread blocking...
input_thread.start() # start the timer thread
# now we'll read the `rasa` STDOUT line by line, forward it to output_buffer and reset
# the timer each time a new line is encountered
for line in proc.stdout:
output_buffer.write(line) # forward the STDOUT line
output_buffer.flush() # flush the output buffer
timer.set() # reset the timer
You can use a similar technique to check for more complex 'expected user input' patterns. There is a whole module called pexpect designed to deal with this type of tasks and I wholeheartedly recommend it if you're willing to give up some flexibility.
Now... all this being said, you are aware that Rasa is built in Python, installs as a Python module and has a Python API, right? Since you're already using Python why would you call it as a subprocess and deal with all this STDOUT/STDIN shenanigans when you can directly run it from your Python code? Just import it and interact with it directly, they even have a very simple example that does exactly what you're trying to do: Rasa Core with minimal Python.
Responding to interactive programs in bash with python subprocess
If you know all answers for external script then you can use subprocess.run() and send all answers (with \n) as one string.
For test I use script which wait for two values USERNAME and COLOR
#example.sh
printf "Hello, who is running this program? "
read -r USERNAME
echo Hi $USERNAME!
printf "What is your favorite color? "
read -r COLOR
echo I like $COLOR too!
And I can use single string "Larry\nBlue\n" to send both values at once.
import subprocess
cmd = ['sh', 'example.sh']
p = subprocess.run(cmd, input="Larry\nBlue\n".encode(), stdout=subprocess.PIPE)
print( p.stdout.decode() )
It sends all at start and external script get first line as first answer, second line as second answer, etc. And when it finish then Python can get all output -
Hello, who is running this program? Hi Larry!
What is your favorite color? I like Blue too!
and you may have to remove elements which you don't need.
But if need to get output from first answer from external script to decide what to send as next answer then it makes problem.
It would need subprocess.Popen() to run and at the same time send input and read output. But there is other problem. If external script send line with \n then you can read it with readline() but if it doesn't send \n then readline() will wait for \n and it will block Python code. The same will be if it send one line but you will try to read 2 lines. Second readline() will block it. So you have to know how many lines it sends.
If you use read() then it will wait for end of data and it will also block Python. If you use read(100) and output will have 100 bytes (or more) then it will read 100 bytes (and rest will wait for next read() - but if it will have 99 bytes then read(100) will block Python.
You will have to know how may bytes to read or you have to know what text it may send and then you may read(1), add it to buffer, and check if buffer has string which you expect. And this method uses expect and pyexpect.
First example with readline()
I use example.sh with \n in printf
printf "Hello, who is running this program? \n"
read -r USERNAME
echo Hi $USERNAME!
printf "What is your favorite color? \n"
read -r COLOR
echo I like $COLOR too!
Because example.sh send with \n so I remove \n
import subprocess
cmd = ['sh', 'example.sh']
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
print('1 >>>', p.stdout.readline().decode().rstrip(), '<<<')
p.stdin.write("Larry!\n".encode())
p.stdin.flush()
print('2 >>>', p.stdout.readline().decode().rstrip(), '<<<')
print('1 >>>', p.stdout.readline().decode().rstrip(), '<<<')
p.stdin.write("Blue!\n".encode())
p.stdin.flush()
print('2 >>>', p.stdout.readline().decode().rstrip(), '<<<')
Result:
1 >>> Hello, who is running this program? <<<
2 >>> Hi Larry!! <<<
1 >>> What is your favorite color? <<<
2 >>> I like Blue! too! <<<
EDIT:
Example in pexpect
Similar to first version - I send all answers and get all output
import pexpect
p = pexpect.spawn('sh script.sh')
p.sendline('Larry')
p.sendline('Blue')
p.expect(pexpect.EOF) # wait for output
print(p.before.decode())
Similar to second example. Pexpect send input as output so it need extra `readline()
import pexpect
p = pexpect.spawn('sh script.sh')
# ---
p.readline()
print('output 1 >>>', p.before.decode(), '<<<')
p.sendline('Larry')
p.readline()
print('input >>>', p.before.decode(), '<<<')
p.readline()
print('output 2 >>>', p.before.decode(), '<<<')
# ---
p.readline()
print('output 1 >>>', p.before.decode(), '<<<')
p.sendline('Blue')
p.readline()
print('input >>>', p.before.decode(), '<<<')
p.readline()
print('output 2 >>>', p.before.decode(), '<<<')
p.expect(pexpect.EOF) # get rest to the end
print('end:', p.before.decode())
Instead of readline() it can use expect(regex) or expect_exact(text)
import pexpect
p = pexpect.spawn('sh script.sh')
# ---
p.expect_exact('\r\n')
print('output 1 >>>', p.before.decode(), '<<<')
p.sendline('Larry')
p.expect_exact('\r\n')
print('input >>>', p.before.decode(), '<<<')
p.expect_exact('\r\n')
print('output 2 >>>', p.before.decode(), '<<<')
# ---
p.expect_exact('\r\n')
print('output 1 >>>', p.before.decode(), '<<<')
p.sendline('Blue')
p.expect_exact('\r\n')
print('input >>>', p.before.decode(), '<<<')
p.expect_exact('\r\n')
print('output 2 >>>', p.before.decode(), '<<<')
p.expect(pexpect.EOF) # get rest to the end
print('end:', p.before.decode())
Python interactive shell not responding to input when run in subprocess
The Problem
Note that you are connecting python to a non-tty standard input, and so it behaves differently from when you just run the command python in your terminal. It instead behaves as if you used the command cat script | python, which means it waits until stdin is closed, and then executes everything as a single script. This behavior is described in the docs:
The interpreter operates somewhat like the Unix shell: when called
with standard input connected to a tty device, it reads and executes
commands interactively; when called with a file name argument or with
a file as standard input, it reads and executes a script from that
file.
Try adding close(in_write_pipe_fd) before reading, and you'll see that it succeeds.
Solution 1: force python to run interactively
To solve your problem, we're gonna need python to ignore the fact it is not run interactively. When running python --help you might notice the flag -i:
-i : inspect interactively after running script; forces a prompt even
if stdin does not appear to be a terminal; also PYTHONINSPECT=x
Sounds good :) Just change your Popen call to:
Popen("python -i", stdin=in_read_pipe_fd, stdout=out_write_pipe_fd,
close_fds=True, shell=True)
And stuff should start working as expected.
Solution 2: pretend to be a terminal
You might have heard of pty, a pseudo-terminal device. It is a feature in a few operating systems that allows you to connect a pipe to a tty driver instead of a terminal emulator, thus, in your case, allowing you write a terminal emulator yourself. You can use the python module pty to open one and connect it to the subprocess instead of a normal pipe. That will trick python to think it is connected to an actual tty device, and will also allow you to emulate Ctrl-C presses, arrow-up/arrow-down, and more.
But that comes with a price - some programs, when connected to a tty, will also alter their output accordingly. For example, in many linux distributions, the grep command colors the matched pattern in the output. If you don't make sure you can handle colors correctly in your program, or configure the tty to declare it doesn't support colors (and other tty features), you'll start getting garbage in some command's outputs.
Small note
I do feel like this might not be the best method to achieve your goal. If you describe it more in detail I might be able to help you think of an alternative :)
How to get input from user and pass it to an interactive command line program triggered by subprocess call in python?
subprocess.call just waits for the process to finish and gives return code, no way to interact with it. if you instead use subprocess.Popen that gives you the ability to communicate with the subprocess while it is running via stdin and stdout
import subprocess, sys
program = subprocess.Popen("python3",
# give us a pipes to coommunicate
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
data = input("input to subprocess")
[out,err] = program.communicate((data+"\n").encode())
print(out.decode())
print(err.decode(), file=sys.stderr)
Doing a bit of input then some output then more input can get messy though since there reading from stdout is blocking so determining when the output has stopped for more input is tricky.
How to pass a value to interactive input in a passive way in Python?
Assuming you are on Linux or wsl, you can just pipe some input through stdin. I.e.,
$ echo $'1\n3\n' | python scripy.py
should produce 4. If you have a file with the same contents, you can do the same using cat
$ cat file.in | python script.py
only if absolutely necessary:
python can also be used to simulate stdin. sys.stdin is simply an instance of the StringIO class. We can override the default sys.stdin with a StringIO class with contents of our choice.
with StringIO(in_string) as f:
tmp = sys.stdin
sys.stdin = f
add()
sys.stdin = tmp
Related Topics
Which Is the Easiest Way to Simulate Keyboard and Mouse on Python
Get a Function Argument's Default Value
Peak-Finding Algorithm for Python/Scipy
How to Activate a Virtualenv Inside Pycharm's Terminal
Python: Best Practice and Securest Way to Connect to MySQL and Execute Queries
Inserting Line at Specified Position of a Text File
Is It Safe to Replace a Self Object by Another Object of the Same Type in a Method
Share Large, Read-Only Numpy Array Between Multiprocessing Processes
Split a String with Unknown Number of Spaces as Separator in Python
When I Catch an Exception, How to Get the Type, File, and Line Number
How to Use Multiprocessing Queue in Python
Python - Is a Dictionary Slow to Find Frequency of Each Character