Using && in Subprocess.Popen for Command Chaining

How to chain Python Popen calls together correctly?

Transferring data between processes is expensive (relative to many simple computations within a process). The kernel amortizes the cost by waiting until a "significant" amount of data is pending—or until the pipe is closed, since obviously there's no point waiting for more data then. This happens between each pair of processes. Your bufsize=0 affects only writes within the parent process, which is why it works only for feeding the first stage of the pipeline.

Some programs provide options to control their output buffering (i.e., they can ask the kernel to transfer data early): Python itself offers python -u. (If it's your Python program, use flush() like g.d.d.c suggested in each process.) Otherwise, there is a (mostly) generic utility unbuffer that might get you immediate responses.

Popen fails if command is a list and contains '&&' or '||'

According to this answer:
Python: subprocess call with shell=False not working

When calling Popen with shell=True you should use a string.
When calling Popen with shell=False you should use a list.

Using '&&' and '||' will only work if you are using shell=True as they require the shell. This means that you can only get this to work using a string as your command, not a list.

If you are getting your command as a list, then you can just do something like:

" ".join(cmd_list)

https://docs.python.org/2/library/subprocess.html#frequently-used-arguments

Special characters to be used in subprocess module

The && logical operator is something that the shell (e.g., bash) understands. However, you're executing git which doesn't understand what && means. Explained another way, you're entering the arguments as if you were typing them into a terminal. subprocess.Popen doesn't invoke the shell by default and instead invokes the exec system call with the specified arguments. So, you're telling it to run git with the arguments add, -A (so far, so good), && (git doesn't understand this), git, -m, and yeah.

To execute this as a shell command, you need to add shell=True to subprocess.Popen. Once you do that, you can just type out the command you want as a single string:

subprocess.Popen('git add -A && git commit -m', shell=true, cwd=path)

running multiple bash commands with subprocess

You have to use shell=True in subprocess and no shlex.split:

import subprocess

command = "echo a; echo b"

ret = subprocess.run(command, capture_output=True, shell=True)

# before Python 3.7:
# ret = subprocess.run(command, stdout=subprocess.PIPE, shell=True)

print(ret.stdout.decode())

returns:

a
b

Popen chaining with command sequence

It's impossible. When cmd is a list, it has different meaning when shell is True. Quoting docs:

If args is a sequence, the first item specifies the command string,
and any additional items will be treated as additional arguments to
the shell itself.

Use ' '.join(shlex.quote(arg) for arg in cmd) (pipes.quote in Python2) when passing list of arguments to Popen with shell=True for expected behavior. Original list won't be mutated, string will be built before passing to function and garbage collected as soon as it's possible.

---

Related Topics