How to Use Subprocess Popen Python

How to use subprocess popen Python

subprocess.Popen takes a list of arguments:

from subprocess import Popen, PIPE

process = Popen(['swfdump', '/tmp/filename.swf', '-d'], stdout=PIPE, stderr=PIPE)
stdout, stderr = process.communicate()

There's even a section of the documentation devoted to helping users migrate from os.popen to subprocess.

Using python with subprocess Popen

To at least really start the subprocess, you have to tell the Popen-object to really communicate.

def run_command(command):
    p = subprocess.Popen(command, shell=True,
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT)
    return p.communicate()

How to use subprocess Popen?

Your command uses pipe |. It requires a shell:

p = subprocess.Popen(command, shell=True)

The command itself as far as I can tell looks ok.

How to use the subprocess Popen.communicate() method?

You're not providing any stdout to the Popen constructor, the default functionality simply writes the output to parent's stdout handle. Hence you're seeing it being printed in your shell.

Quoting from Popen's docs:

With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent.

---

To populate stdout in resulting tuple use subprocess.PIPE as stdout.

Quoting from docs:

To get anything other than None in the result tuple, you need to give
stdout=PIPE and/or stderr=PIPE too.

>>> import subprocess
>>> p = subprocess.Popen(["echo", "hello"], stdout=subprocess.PIPE)
>>> p.communicate()
('hello\n', None)

How to use subprocess.Popen() instead of os.popen() in Python?

Just use subprocess.Popen to create a new process, with its stdout redirected to a PIPE, in text mode and then read its stdout.

Python version >= 3.6

from subprocess import Popen, PIPE
with Popen(f'ps {pid}'.split(), stdout=PIPE, text=True) as proc:
    pinfo = proc.stdout.readlines()

As you have requested for python2.7, I have given the code. But remember that python2.7 has reached EOL and you should be using python3.x

Python version = 2.7

from subprocess import Popen, PIPE
proc = Popen(f'ps {pid}'.split(), stdout=PIPE)
pinfo = proc.stdout.readlines()

Refer subprocess documentation for more information

What is the difference between subprocess.popen and subprocess.run

subprocess.run() was added in Python 3.5 as a simplification over subprocess.Popen when you just want to execute a command and wait until it finishes, but you don't want to do anything else in the mean time. For other cases, you still need to use subprocess.Popen.

The main difference is that subprocess.run() executes a command and waits for it to finish, while with subprocess.Popen you can continue doing your stuff while the process finishes and then just repeatedly call Popen.communicate() yourself to pass and receive data to your process. Secondly, subprocess.run() returns subprocess.CompletedProcess.

subprocess.run() just wraps Popen and Popen.communicate() so you don't need to make a loop to pass/receive data or wait for the process to finish.

Check the official documentation for info on which params subprocess.run() pass to Popen and communicate().

Run Python script within Python by using `subprocess.Popen` in real time

Last night I've set out to do this using a pipe:

import os
import subprocess

with open("test2", "w") as f:
    f.write("""import time
print('start')
time.sleep(2)
print('done')""")

(readend, writeend) = os.pipe()

p = subprocess.Popen(['python3', '-u', 'test2'], stdout=writeend, bufsize=0)
still_open = True
output = ""
output_buf = os.read(readend, 1).decode()
while output_buf:
    print(output_buf, end="")
    output += output_buf
    if still_open and p.poll() is not None:
        os.close(writeend)
        still_open = False
    output_buf = os.read(readend, 1).decode()

Forcing buffering out of the picture and reading one character at the time (to make sure we do not block writes from the process having filled a buffer), closing the writing end when process finishes to make sure read catches the EOF correctly. Having looked at the subprocess though that turned out to be a bit of an overkill. With PIPE you get most of that for free and I ended with this which seems to work fine (call read as many times as necessary to keep emptying the pipe) with just this and assuming the process finished, you do not have to worry about polling it and/or making sure the write end of the pipe is closed to correctly detect EOF and get out of the loop:

p = subprocess.Popen(['python3', '-u', 'test2'],
                     stdout=subprocess.PIPE, bufsize=1,
                     universal_newlines=True)
output = ""
output_buf = p.stdout.readline()
while output_buf:
    print(output_buf, end="")
    output += output_buf
    output_buf = p.stdout.readline()

This is a bit less "real-time" as it is basically line buffered.

Note: I've added -u to you Python call, as you need to also make sure your called process' buffering does not get in the way.

How can I use subprocess.Popen in a for loop?

Per @jasonharper's suggestion, I needed to make a new open file object for each iteration.

input_file = Path("input.json")

for thread in threads:
    new_env = default_env
    new_env["OMP_NUM_THREADS"] = str(thread)
    print("Starting run for {} threads".format(thread))
    process = subprocess.Popen(
        benchmark_command, env=new_env, stdin=input_file.open(), stdout=subprocess.PIPE)
    lines = []
    for line in process.stdout:
        print(line.decode(), end='')
    results.append(Result(lines, thread))
    print("Completed run for {} threads".format(thread))

---

Related Topics