Using Python Subprocess.Call() to Launch an Ncurses Process

Using Python subprocess.call() to launch an ncurses process

To run curses programs under Python I'd recommend that you use pexpect.

For example here's a simple program that starts a copy of vim, add some text, escape to command mode, issue a :w command, and then interact with the user (allowing him or her to continue editing or whatever). Then the control returns to Python:

#!/usr/bin/env python
import pexpect
child = pexpect.spawn("/usr/bin/vim")
child.send('a\n\nThis is another test.')
child.send('\x1b')
child.send(':w! test.txt\n')
child.interact()

You can also pass arguments (such as escape character, and filter functions for input and output) to the interact method. But those get a bit tricky. (On the other hand they then become your custom keyboard macro system interposed between users and the application being run under the .spawn()).

(BTW: you can send your desired sequences of keystrokes into this ct-ng dialog/menu ... it's just a matter of figuring out what those sequences need to be for your terminal settings. For example on my iTerm under MacOS X running with TERM=xterm-256color a "down arrow" cursor movement comes out as ^[[B ([Esc][Bracket][B]). That would be '\x1b[B' as a Python string literal).

How to call an ncurses based application using subprocess module in PyCharm IDE?

I don't know PyCharm or TABARI specifically, but from the error message it sounds like PyCharm is executing your code without connecting it to a terminal. Possibly it does this so it can just collect program output and display it in a GUI window, or because the authors don't feel like it's very clean to launch a terminal emulator like xterm and run your code inside that.

From some of the other questions around here, it sounds like there isn't any really good way to make PyCharm provide a terminal-emulation environment when running your code. There are some suggestions on this question, but they don't sound very satisfactory.

The path of least resistance is probably just to run your program from the terminal each time. If that's unacceptable, you could have your code check to see if stdin is a terminal (os.isatty(0)), and if not, explicitly launch a terminal emulator like xterm and re-invoke your code under that. Or, if you don't actually need to interact with the subprocess while it runs, you could allocate your own pseudoterminal master/slave pair and run the code connected to the slave. These things are all more complicated than they probably should be, and a full explanation of all of it would take enough text to fill a whole manual, but here are some good resources:

• Wikipedia entry on Pseudo Terminals, for some very general background
• man page for xterm(1), for info on how to launch with a particular command instead of your shell
• man page for pty(7)- explains the mechanics of interacting with pty/tty devices
• the Python pty module, in case you want to make a pseudoterminal master/slave pair and interact with it from plain Python
• an explanation from an old-ish Linux Kernel manual regarding how process groups and sessions relate to terminal ownership
• an excerpt from Advanced Programming in the UNIX® Environment: Second Edition
  By W. Richard Stevens, Stephen A. Rago with some more info about terminal control

Running system commands in Python using curses and panel, and come back to previous menu

You really have two choices. One you can leave curses mode, execute your program, then resume curses. Two, you can execute your program asynchronously, parse its output and write it to the screen.

The good news on the first option is that you don't actually need to write any fancy save_state / load_state methods for the ui. Curses does this for you. Here's a simple example to show my point

import curses, time, subprocess

class suspend_curses():
    """Context Manager to temporarily leave curses mode"""

    def __enter__(self):
        curses.endwin()

    def __exit__(self, exc_type, exc_val, tb):
        newscr = curses.initscr()
        newscr.addstr('Newscreen is %s\n' % newscr)
        newscr.refresh()
        curses.doupdate()

def main(stdscr):
    stdscr.addstr('Stdscreen is %s\n' % stdscr)
    stdscr.refresh()
    time.sleep(1)

    with suspend_curses():
        subprocess.call(['ls'])
        time.sleep(1)

    stdscr.refresh()
    time.sleep(5)

curses.wrapper(main)

If you run the example, you will notice that the screen created by curses.wrapper and the one created in curses.initscr when resuming are the same object. That is, the window returned by curses.initscr is a singleton. This lets us exit curses and resume like above without having to update each widget's self.screen references each time.

The second option is much more involved but also much more flexible. The following is just to represent the basic idea.

class procWidget():
    def __init__(self, stdscr):
        # make subwindow / panel
        self.proc = subprocess.Popen(my_args, stdout=subprocess.PIPE)

    def update(self):
        data = self.proc.stdout.readline()
        # parse data as necessary
        # call addstr() and refresh()

Then somewhere in your program you will want to call update on all your procWidgets on a timer. This gives you the option of making your subwindow any size/place so you can have as many procWidgets as will fit. You will have to add some handling for when the process terminates and other similar events of course.

Real-time colour refreshing output from python's subprocess.Popen()

Well, the right words to Google were ncurses/curses, which immediately led me to this answer.

The following couple of lines worked for me like a charm:

import pexpect
child = pexpect.spawn(<command_line_str>)
child.interact()

How to kill a Windows subprocess in Python when it expects a key but simply doesn't react to it through stdin?

There are multiple ways in which a python script can communicate with a subprocess when it comes to keypresses.

• pywin32
• pynput
• pyautogui
• ctypes + user32.dll

Examples

PyWin32

(credits to @john-hen -> inspired from https://stackoverflow.com/a/8117562/858565)

Package: https://pypi.org/project/pywin32/

import win32console

def make_keydown_input(c):
    input_record = win32console.PyINPUT_RECORDType(win32console.KEY_EVENT)
    input_record.KeyDown = 1
    input_record.RepeatCount = 1
    input_record.Char = c
    return input_record

console_handler = win32console.GetStdHandle(win32console.STD_INPUT_HANDLE)

q_keydown = make_keydown_input('q')
console_handler.WriteConsoleInput([q_keydown])

pynput

Package: https://pypi.org/project/pynput/

import pynput

keyb_controller = pynput.keyboard.Controller()

keyb_controller.press('q')

pyautogui

Package: https://pypi.org/project/PyAutoGUI/

Examples: https://pyautogui.readthedocs.io/en/latest/keyboard.html

This is probably the simplest method.

import pyautogui

pyautogui.press('q')

ctypes + user32.dll

Please refer to this answer: https://stackoverflow.com/a/13615802/858565

In the end you just have to call:

PressKey(0x51)

where 0x51 is the hex keycode for q according to (https://learn.microsoft.com/en-us/windows/win32/inputdev/virtual-key-codes)

Observations

Notice that all of these methods will apply the keypress in the context of the program that calls it. I have not tried it with multiprocessing.Process, perhaps it will react differently.

Probably to have a good isolated context one would have to run a specific subprocess in an isolated python script that simply receives messages from outside like via a socket.

Dynamic output from python subprocess module

I know this is a duplicate, but I can't find any relevant threads about this now. All i get is output.communicate().

So here's a snippet that might be useful:

import subprocess
cmd = ['ngrok', 'http', '5000']
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

while process.poll() is None:
    print(process.stdout.readline())
print(process.stdout.read())
process.stdout.close()

This would output anything the process outputs, through your script into your output. It does so by looking for a newline character before outputting.

This piece of code would work, if it weren't for the fact that ngrok uses ncurses and/or hogs the output to it's own user/thread much like when SSH asks for a password when you do ssh user@host.

process.poll() checks if the process has an exit-code (if it's dead), if not, it continues to loop and print anything from the process's stdout.

There's other (better) ways to go about it, but this is the bare minimum I can give you without it being complicated really fast.

For instance, process.stdout.read() could be used in junction with select.select() to achieve better buffered output where new-lines are scares. Because if a \n never comes, the above example might hang your entire application.

There's a lot of buffer-traps here that you need to be aware of before you do manual things like this. Otherwise, use process.communicate() instead.

Edit: To get around the hogging/limitation of I/O used by ngrok, you could use pty.fork() and read the childs stdout via the os.read module:

#!/usr/bin/python

## Requires: Linux
## Does not require: Pexpect

import pty, os
from os import fork, waitpid, execv, read, write, kill

def pid_exists(pid):
    """Check whether pid exists in the current process table."""
    if pid < 0:
        return False
    try:
        kill(pid, 0)
    except (OSError, e):
        return e.errno == errno.EPERMRM
    else:
        return True

class exec():
    def __init__(self):
        self.run()

    def run(self):
        command = [
                '/usr/bin/ngrok',
                'http',
                '5000'
        ]

        # PID = 0 for child, and the PID of the child for the parent    
        pid, child_fd = pty.fork()

        if not pid: # Child process
            # Replace child process with our SSH process
            execv(command[0], command)

        while True:
            output = read(child_fd, 1024)
            print(output.decode('UTF-8'))
            lower = output.lower()

            # example input (if needed)
            if b'password:' in lower:
                write(child_fd, b'some response\n')
        waitpid(pid, 0)

exec()

There's still a problem here, and I'm not quite sure what or why that is.

I'm guessing the process is waiting for a signal/flush some how.

The problem is that it's only printing the first "setup data" of ncurses, meaning it wips the screen and sets up the background-color.

But this would give you the output of the process at least. replacing print(output.decode('UTF-8')) would show you what that output is.

---

Related Topics