What's the Difference Between Subprocess Popen and Call (How to Use Them)

What's the difference between subprocess Popen and call (how can I use them)?

There are two ways to do the redirect. Both apply to either subprocess.Popen or subprocess.call.

Set the keyword argument shell = True or executable = /path/to/the/shell and specify the command just as you have it there.
Since you're just redirecting the output to a file, set the keyword argument
```
stdout = an_open_writeable_file_object
```
where the object points to the output file.

subprocess.Popen is more general than subprocess.call.

Popen doesn't block, allowing you to interact with the process while it's running, or continue with other things in your Python program. The call to Popen returns a Popen object.

call does block. While it supports all the same arguments as the Popen constructor, so you can still set the process' output, environmental variables, etc., your script waits for the program to complete, and call returns a code representing the process' exit status.

returncode = call(*args, **kwargs)

is basically the same as calling

returncode = Popen(*args, **kwargs).wait()

call is just a convenience function. It's implementation in CPython is in subprocess.py:

def call(*popenargs, timeout=None, **kwargs):
    """Run command with arguments.  Wait for command to complete or
    timeout, then return the returncode attribute.

    The arguments are the same as for the Popen constructor.  Example:

    retcode = call(["ls", "-l"])
    """
    with Popen(*popenargs, **kwargs) as p:
        try:
            return p.wait(timeout=timeout)
        except:
            p.kill()
            p.wait()
            raise

As you can see, it's a thin wrapper around Popen.

What is the difference between subprocess.popen and subprocess.run

subprocess.run() was added in Python 3.5 as a simplification over subprocess.Popen when you just want to execute a command and wait until it finishes, but you don't want to do anything else in the mean time. For other cases, you still need to use subprocess.Popen.

The main difference is that subprocess.run() executes a command and waits for it to finish, while with subprocess.Popen you can continue doing your stuff while the process finishes and then just repeatedly call Popen.communicate() yourself to pass and receive data to your process. Secondly, subprocess.run() returns subprocess.CompletedProcess.

subprocess.run() just wraps Popen and Popen.communicate() so you don't need to make a loop to pass/receive data or wait for the process to finish.

Check the official documentation for info on which params subprocess.run() pass to Popen and communicate().

What difference between subprocess.call() and subprocess.Popen() makes PIPE less secure for the former?

call() is just Popen().wait() (± error handling).

You should not use stdout=PIPE with call() because it does not read from the pipe and therefore the child process will hang as soon as it fills the corresponding OS pipe buffer. Here's a picture that shows how data flows in command1 | command2 shell pipeline:

pipe/stdio buffers

It does not matter what your Python version is -- the pipe buffer (look at the picture) is outside of your Python process. Python 3 does not use C stdio but it affects only the internal buffering. When the internal buffer is flushed the data goes into the pipe. If command2 (your parent Python program) does not read from the pipe then command1 (the child process e.g., started by call()) will hang as soon as the pipe buffer is full (pipe_size = fcntl(p.stdout, F_GETPIPE_SZ) ~65K on my Linux box (max value is /proc/sys/fs/pipe-max-size ~1M)).

You may use stdout=PIPE if you read from the pipe later e.g., using Popen.communicate() method. You could also read from process.stdout (the file object that represents the pipe) directly.

Difference between subprocess Popen/call/check_output

call and check_output (along with check_call) are just utility functions which call Popen under the hood.

call returns the exit code of child process
check_call raises CalledProcessError error if exit code was non zero
check_output same as above but also returns output.

The difference between readlines and communicate is that readlines is simply a function made on the buffer (stdout) while communicate is a method of process class so it can handle different exceptions, you can pass input in it, and it waits for the process to finish.

What's the difference between subprocess.Popen(echo $HOME... and subprocess.Popen([echo, $HOME]

The first argument to subprocess.Popen() tells the system what to run.

When it is a list, you need to use shell=False. It coincidentally happens to work as you hope in Windows; but on Unix-like platforms, you are simply passing in a number of arguments which will typically get ignored. Effectively,

/bin/sh -c 'echo' '$HOME'

which simply causes the second argument to not be used for anything (where I use single quotes to emphasize that these are just static strings).

In my humble opinion, Python should throw an error in this case. On Windows, too. This is an error which should be caught and reported.

(In the opposite case, where shell=False is specified but the string you pass in is not the name of a valid command, you will get an error eventually anyway, and it makes sense if you have even a vague idea of what's going on.)

If you really know what you are doing, you could cause the first argument to access subsequent arguments; for example

/bin/sh -c 'printf "%s\n" "$@"' 'ick' 'foo' 'bar' 'baz'

would print foo, bar, and baz on separate lines. (The "zeroth" argument - here, 'ick' - is used to populate $0.) But this is just an obscure corollary; don't try to use this for anything.

As a further aside, you should not use subprocess.Popen() if you just want a command to run. The subprocess.run() documentation tells you this in some more detail. With text=True you get a string instead of bytes.

result = subprocess.run('echo "$HOME"', shell=True,
    text=True, capture_output=True, check=True)
print(result.stdout, result.stderr)

And of course, os.environ['HOME'] lets you access the value of $HOME from within Python. This also allows you to avoid shell=True which you usually should if you can.

Is there a difference between subprocess.call() and subprocess.Popen.communicate()?

Like the documentation already tells you, you want to avoid Popen whenever you can.

The subprocess.check_output() function in Python 2.7 lets you retrieve the output from the subprocess, but otherwise works basically like check_call() (which in turn differs from call only in that it will raise an exception if the subprocess fails, which is something you usually want and need).

The case for Popen is that it enables you to build new high-level functions like these if you need to (like also then subprocess.run() in Python 3.5+ which is rather more versatile, and basically subsumes the functionality of all the above three). They all use Popen under the hood, but it is finicky and requires you to take care of several details related to managing the subprocess object if you use it directly; those higher-level functions already do those things for you.

Common cases where you do need Popen is if you want the subprocess to run in parallel with your main Python script. There is no simple library function which does this for you.

Subprocess.Popen vs .call: What is the correct way to call a C-executable from shell script using python where all 6 jobs can run in parallel

You tried to simulate multiprocessing with subprocess.Popen() which does not work like you want: the output is blocked after a while unless you consume it, for instance with communicate() (but this is blocking) or by reading the output, but with 6 concurrent handles in a loop, you are bound to get deadlocks.

The best way is run the subprocess.call lines in separate threads.

There are several ways to do it. Small simple example with locking:

import threading,time

lock=threading.Lock()
def func1(a,b,c):
    lock.acquire()
    print(a,b,c)
    lock.release()
    time.sleep(10)

tl=[]
t = threading.Thread(target=func1,args=[1,2,3])
t.start()
tl.append(t)
t=threading.Thread(target=func1,args=[4,5,6])
t.start()
tl.append(t)

# wait for all threads to complete (if you want to wait, else
# you can skip this loop)

for t in tl:
    t.join()

I took the time to create an example more suitable to your needs:

2 threads executing a command and getting the output, then printing it within a lock to avoid mixup. I have used check_output method for this. I'm using windows, and I list C and D drives in parallel.

import threading,time,subprocess

lock=threading.Lock()

def func1(runrescompare,rescompare_dir):
    resrun_proc = subprocess.check_output(runrescompare, shell=True, cwd=rescompare_dir, stderr=subprocess.PIPE, universal_newlines=True)
    lock.acquire()
    print(resrun_proc)
    lock.release()

tl=[]
t=threading.Thread(target=func1,args=["ls","C:/"])
t.start()
tl.append(t)
t=threading.Thread(target=func1,args=["ls","D:/"])
t.start()
tl.append(t)

# wait for all threads to complete (if you want to wait, else
# you can skip this loop)

for t in tl:
    t.join()

Difference between subprocess.Popen and os.system

If you check out the subprocess section of the Python docs, you'll notice there is an example of how to replace os.system() with subprocess.Popen():

sts = os.system("mycmd" + " myarg")

...does the same thing as...

sts = Popen("mycmd" + " myarg", shell=True).wait()

The "improved" code looks more complicated, but it's better because once you know subprocess.Popen(), you don't need anything else. subprocess.Popen() replaces several other tools (os.system() is just one of those) that were scattered throughout three other Python modules.

If it helps, think of subprocess.Popen() as a very flexible os.system().