Running an Interactive Command from Within Python

Running interactive program from within python

You need to keep interacting with your subprocess - at the moment once you pick the output from your subprocess you're pretty much done as you close its STDOUT stream.

Here is the most rudimentary way to continue user input -> process output cycle:

import subprocess
import sys
import time

if __name__ == "__main__":  # a guard from unintended usage
    input_buffer = sys.stdin  # a buffer to get the user input from
    output_buffer = sys.stdout  # a buffer to write rasa's output to
    proc = subprocess.Popen(["path/to/rasa", "arg1", "arg2", "etc."],  # start the process
                            stdin=subprocess.PIPE,  # pipe its STDIN so we can write to it
                            stdout=output_buffer, # pipe directly to the output_buffer
                            universal_newlines=True)
    while True:  # run a main loop
        time.sleep(0.5)  # give some time for `rasa` to forward its STDOUT
        print("Input: ", end="", file=output_buffer, flush=True)  # print the input prompt
        print(input_buffer.readline(), file=proc.stdin, flush=True)  # forward the user input

You can replace input_buffer with a buffer coming from your remote user(s) and output_buffer with a buffer that forwards the data to your user(s) and you'll get essentially what you're looking for - the sub-process will be getting the input directly from the user (input_buffer) and print its output to the user (output_buffer).

If you need to perform other tasks while all this is running in the background, just run everything under the if __name__ == "__main__": guard in a separate thread, and I'd suggest adding a try..except block to pick up KeyboardInterrupt and exit gracefully.

But... soon enough you'll notice that it doesn't exactly work properly all the time - if it takes longer than half a second of wait for rasa to print its STDOUT and enter the wait for STDIN stage, the outputs will start to mix. This problem is considerably more complex than you might expect. The main issue is that STDOUT and STDIN (and STDERR) are separate buffers and you cannot know when a subprocess is actually expecting something on its STDIN. This means that without a clear indication from the subprocess (like you have the \r\n[path]> in Windows CMD prompt on its STDOUT for example) you can only send data to the subprocesses STDIN and hope it will be picked up.

Based on your screenshot, it doesn't really give a distinguishable STDIN request prompt because the first prompt is ... :\n and then it waits for STDIN, but then once the command is sent it lists options without an indication of its end of STDOUT stream (technically making the prompt just ...\n but that would match any line preceding it as well). Maybe you can be clever and read the STDOUT line by line, then on each new line measure how much time has passed since the sub-process wrote to it and once a threshold of inactivity is reached assume that rasa expects input and prompt the user for it. Something like:

import subprocess
import sys
import threading

# we'll be using a separate thread and a timed event to request the user input
def timed_user_input(timer, wait, buffer_in, buffer_out, buffer_target):
    while True:  # user input loop
        timer.wait(wait)  # wait for the specified time...
        if not timer.is_set():  # if the timer was not stopped/restarted...
            print("Input: ", end="", file=buffer_out, flush=True)  # print the input prompt
            print(buffer_in.readline(), file=buffer_target, flush=True)  # forward the input
        timer.clear()  # reset the 'timer' event

if __name__ == "__main__":  # a guard from unintended usage
    input_buffer = sys.stdin  # a buffer to get the user input from
    output_buffer = sys.stdout  # a buffer to write rasa's output to
    proc = subprocess.Popen(["path/to/rasa", "arg1", "arg2", "etc."],  # start the process
                            stdin=subprocess.PIPE,  # pipe its STDIN so we can write to it
                            stdout=subprocess.PIPE,  # pipe its STDIN so we can process it
                            universal_newlines=True)
    # lets build a timer which will fire off if we don't reset it
    timer = threading.Event()  # a simple Event timer
    input_thread = threading.Thread(target=timed_user_input,
                                    args=(timer,  # pass the timer
                                          1.0,  # prompt after one second
                                          input_buffer, output_buffer, proc.stdin))
    input_thread.daemon = True  # no need to keep the input thread blocking...
    input_thread.start()  # start the timer thread
    # now we'll read the `rasa` STDOUT line by line, forward it to output_buffer and reset
    # the timer each time a new line is encountered
    for line in proc.stdout:
        output_buffer.write(line)  # forward the STDOUT line
        output_buffer.flush()  # flush the output buffer
        timer.set()  # reset the timer

You can use a similar technique to check for more complex 'expected user input' patterns. There is a whole module called pexpect designed to deal with this type of tasks and I wholeheartedly recommend it if you're willing to give up some flexibility.

Now... all this being said, you are aware that Rasa is built in Python, installs as a Python module and has a Python API, right? Since you're already using Python why would you call it as a subprocess and deal with all this STDOUT/STDIN shenanigans when you can directly run it from your Python code? Just import it and interact with it directly, they even have a very simple example that does exactly what you're trying to do: Rasa Core with minimal Python.

How do I execute a program or call a system command?

Use the subprocess module in the standard library:

import subprocess
subprocess.run(["ls", "-l"])

The advantage of subprocess.run over os.system is that it is more flexible (you can get the stdout, stderr, the "real" status code, better error handling, etc...).

Even the documentation for os.system recommends using subprocess instead:

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.

On Python 3.4 and earlier, use subprocess.call instead of .run:

subprocess.call(["ls", "-l"])

Executing Interactive shell script in python

Your code is working, but since you mention stdout=subprocess.PIPE the content is going to stdout variable you defined in stdout,stderr = proc.communicate(). Remove stdout=subprocess.PIPE argument from your Popen() call and you will see the output.

Alternatively, you should be using subprocess.check_call() as:

subprocess.check_call(shlex.split(cmd))

Interactive input/output using Python

Two solutions for this issue on Linux:

First one is to use a file to write the output to, and read from it simultaneously:

from subprocess import Popen, PIPE

fw = open("tmpout", "wb")
fr = open("tmpout", "r")
p = Popen("./a.out", stdin = PIPE, stdout = fw, stderr = fw, bufsize = 1)
p.stdin.write("1\n")
out = fr.read()
p.stdin.write("5\n")
out = fr.read()
fw.close()
fr.close()

Second, as J.F. Sebastian offered, is to make p.stdout and p.stderr pipes non-blocking using fnctl module:

import os
import fcntl
from subprocess import Popen, PIPE  
def setNonBlocking(fd):
    """
    Set the file description of the given file descriptor to non-blocking.
    """
    flags = fcntl.fcntl(fd, fcntl.F_GETFL)
    flags = flags | os.O_NONBLOCK
    fcntl.fcntl(fd, fcntl.F_SETFL, flags)

p = Popen("./a.out", stdin = PIPE, stdout = PIPE, stderr = PIPE, bufsize = 1)
setNonBlocking(p.stdout)
setNonBlocking(p.stderr)

p.stdin.write("1\n")
while True:
    try:
        out1 = p.stdout.read()
    except IOError:
        continue
    else:
        break
out1 = p.stdout.read()
p.stdin.write("5\n")
while True:
    try:
        out2 = p.stdout.read()
    except IOError:
        continue
    else:
        break

running python script in interactive python prompt and keep the variables?

Try the following for Python 2.x:

>>> execfile('first.py')

For Python 3.x, try this:

>>> exec(open("./first.py").read())

The variables should then be available to you.

run command in interactive opend bash with python

This can be done with pure pipes, but it's going to be hard. And even harder if you use os.popen() instead of using subprocess.

The right way to script an interactive program is to use a higher-level library that's designed to make it easy, like pexpect. Then you just write something like:

import pexpect
child = pexpect.spawn('vlc -I rc --novideo --noaudio --rc-fake-tty -q  udp://@1.2.3.4:1234')
child.expect('>')
child.sendline('info')
response = child.before

However, a much better solution is to not run VLC in interactive mode; just run it in batch mode and pass it commands. Going out of your way to have it treat your input as a TTY just so you can try to figure out how to act like a human at a TTY is making things harder for no good reason.

Or, even better, use libVLC instead. As you can see from that link, there are Python bindings for it.

If you really want to do it interactively, and you want to do it manually over pipes, you will have to be very careful. If you don't mind just deadlocking on any unexpected results, you can do something like this:

import subprocess
child = subprocess.Popen(['vlc', '-I', 'rc', '--novideo', '--noaudio',
                          '--rc-fake-tty', '-q', 'udp://@1.2.3.4:1234'],
                         stdin=PIPE, stdout=PIPE)
def split_on_prompts():
    rbuf = ''
    while True:
        newbuf = child.stdout.read()
        rbuf += newbuf
        out, prompt, rest = rbuf.partition('\n>')
        if prompt:
            yield out
            rbuf = rest
        if not newbuf:
            yield rest
            return
output = split_on_prompts()
banner = next(output)
child.stdin.write('info\n')
response = next(output)
# etc.

As you can see, this is a lot less fun.

And if you insist on using os.open instead even though it's deprecated and even more painful to use, you obviously can't write to it if you open the pipe in the default 'r' mode, just like any other file-like object, and of course tacking .read() on the end means you don't even have the popen object anymore, you just stored the first buffer it gave you and then leaked the handle. If you change that to open in 'r+' mode, if that works on your platform, and you store the popen object itself, you can use it similarly to the subprocess.Popen object above, using child.write and child.read instead of child.stdin.writeandchild.stdout.read`.

Can I run an interactive command?

What you want to do is add

from sys import stderr,stdout

and then replace

return self.exec_command(cmd)

with

return self.exec_command(cmd,stdout=stdout,stderr=stderr)

(Tested with Waf 1.9.11 and Python 2.7.6)

Responding to interactive programs in bash with python subprocess

If you know all answers for external script then you can use subprocess.run() and send all answers (with \n) as one string.

For test I use script which wait for two values USERNAME and COLOR

#example.sh
printf "Hello, who is running this program? "

read -r USERNAME

echo Hi $USERNAME!

printf "What is your favorite color? "

read -r COLOR

echo I like $COLOR too!

And I can use single string "Larry\nBlue\n" to send both values at once.

import subprocess

cmd = ['sh', 'example.sh']

p = subprocess.run(cmd, input="Larry\nBlue\n".encode(), stdout=subprocess.PIPE)

print( p.stdout.decode() )

It sends all at start and external script get first line as first answer, second line as second answer, etc. And when it finish then Python can get all output -

Hello, who is running this program? Hi Larry!
What is your favorite color? I like Blue too!

and you may have to remove elements which you don't need.

But if need to get output from first answer from external script to decide what to send as next answer then it makes problem.

It would need subprocess.Popen() to run and at the same time send input and read output. But there is other problem. If external script send line with \n then you can read it with readline() but if it doesn't send \n then readline() will wait for \n and it will block Python code. The same will be if it send one line but you will try to read 2 lines. Second readline() will block it. So you have to know how many lines it sends.

If you use read() then it will wait for end of data and it will also block Python. If you use read(100) and output will have 100 bytes (or more) then it will read 100 bytes (and rest will wait for next read() - but if it will have 99 bytes then read(100) will block Python.

You will have to know how may bytes to read or you have to know what text it may send and then you may read(1), add it to buffer, and check if buffer has string which you expect. And this method uses expect and pyexpect.

First example with readline()

I use example.sh with \n in printf

printf "Hello, who is running this program? \n"

read -r USERNAME

echo Hi $USERNAME!

printf "What is your favorite color? \n"

read -r COLOR

echo I like $COLOR too!

Because example.sh send with \n so I remove \n

import subprocess

cmd = ['sh', 'example.sh']

p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    
print('1 >>>', p.stdout.readline().decode().rstrip(), '<<<')
p.stdin.write("Larry!\n".encode())
p.stdin.flush()
print('2 >>>', p.stdout.readline().decode().rstrip(), '<<<')

print('1 >>>', p.stdout.readline().decode().rstrip(), '<<<')
p.stdin.write("Blue!\n".encode())
p.stdin.flush()
print('2 >>>', p.stdout.readline().decode().rstrip(), '<<<')

Result:

1 >>> Hello, who is running this program? <<<
2 >>> Hi Larry!! <<<
1 >>> What is your favorite color? <<<
2 >>> I like Blue! too! <<<

EDIT:

Example in pexpect

Similar to first version - I send all answers and get all output

import pexpect    

p = pexpect.spawn('sh script.sh')

p.sendline('Larry')
p.sendline('Blue')

p.expect(pexpect.EOF)  # wait for output

print(p.before.decode())

Similar to second example. Pexpect send input as output so it need extra `readline()

import pexpect    

p = pexpect.spawn('sh script.sh')

# ---
    
p.readline()
print('output 1 >>>', p.before.decode(), '<<<')

p.sendline('Larry')

p.readline()
print('input    >>>', p.before.decode(), '<<<')
p.readline()
print('output 2 >>>', p.before.decode(), '<<<')

# ---
    
p.readline()
print('output 1 >>>', p.before.decode(), '<<<')

p.sendline('Blue')

p.readline()
print('input    >>>', p.before.decode(), '<<<')
p.readline()
print('output 2 >>>', p.before.decode(), '<<<')

p.expect(pexpect.EOF)  # get rest to the end
print('end:', p.before.decode())

Instead of readline() it can use expect(regex) or expect_exact(text)

import pexpect    

p = pexpect.spawn('sh script.sh')

# ---
    
p.expect_exact('\r\n')
print('output 1 >>>', p.before.decode(), '<<<')

p.sendline('Larry')

p.expect_exact('\r\n')
print('input    >>>', p.before.decode(), '<<<')
p.expect_exact('\r\n')
print('output 2 >>>', p.before.decode(), '<<<')

# ---
    
p.expect_exact('\r\n')
print('output 1 >>>', p.before.decode(), '<<<')

p.sendline('Blue')

p.expect_exact('\r\n')
print('input    >>>', p.before.decode(), '<<<')
p.expect_exact('\r\n')
print('output 2 >>>', p.before.decode(), '<<<')

p.expect(pexpect.EOF)  # get rest to the end
print('end:', p.before.decode())