Subprocess.Call Using String VS Using List

subprocess.call using string vs using list

subprocess's rules for handling the command argument are actually a bit complex.

Generally speaking, to run external commands, you should use shell=False and pass the arguments as a sequence. Use shell=True only if you need to use shell built-in commands or specific shell syntax; using shell=True correctly is platform-specific as detailed below.

From the docs:

args should be a sequence of program arguments or else a single string. By default, the program to execute is the first item in args if args is a sequence. If args is a string, the interpretation is platform-dependent and described below. See the shell and executable arguments for additional differences from the default behavior. Unless otherwise stated, it is recommended to pass args as a sequence.... If shell is True, it is recommended to pass args as a string rather than as a sequence.

With shell=False:

On Unix, if args is a string, the string is interpreted as the name or path of the program to execute. However, this can only be done if not passing arguments to the program.

On Windows, if args is a sequence, it will be converted to a string in a manner described in Converting an argument sequence to a string on Windows. This is because the underlying CreateProcess() operates on strings.

With shell=True:

On Unix with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself.

On Windows with shell=True, the COMSPEC environment variable specifies the default shell. The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable.

(all emphasis mine)

Difference between whole string command and list of strings in popen

The second should not have a shell=True parameter. Instead, it should be:
subprocess.Popen(['pgrep', '-f', '"\./run"'], stdout=subprocess.PIPE).wait().

The shell parameter sets whether or not to execute the command in a separate shell. That is, if a new shell should be spawned just to execute the command, which must be interpreted by the shell before it can be run.

When providing a list of strings, however, this does not spawn a second shell, and thus is (minimally) faster. It is also better to use for processing variable input, because it avoids string interpolation.

See: https://stackoverflow.com/a/15109975/1730261

python subprocess run works with single string but not list of strings

Split each parameter with its value, like so:

step1_cmd = [exe, 
             "--step",
             "1",
             "--p",
             str(pheno_fp),  # if it isn't a string already
             "--b",
             "1000",
             "--o",
             str(construction_fp) + "dpw_leaveout"
]

Because when passing a list of parameters, each part is separated with a space, both options and their values

Why does subprocess use a list instead of a string with spaces by default?

TL;DR Using the list bypasses the shell so that you don't need to worry about the shell interpreting a dynamically constructed command line in ways you did not intend.

---

Suppose you have a really simple command: echo foo. Here it is, using both a string and a list:

Popen("echo foo", shell=True)
Popen(["echo", "foo"])

Not much difference yet. Now suppose the argument contains quotes to protect whitespace and/or a shell pattern, echo "foo * bar":

Popen("echo \"foo   *    bar\"", shell=True)
Popen(["echo", "foo   *    bar"])

Yes, I could have used single quotes to avoid needing to escape the double quotes, but you can see the list form is starting to have an advantage. Now imagine I don't have a literal argument for the command, but that it is stored in a variable. Now which do you want to use...

This?

Popen('echo "%s"' % (x,), shell=True)

or this?

Popen(["echo", x])

If you answered "the first one", here's the value of x:

x = "\";rm -rf \""

The command you just executed was echo ""; rm -rf/"". You needed to make sure any special characters in the value of x were first escaped before incorporating it into the string you are building to pass to the shell.

Or you just use a list and avoid the shell altogether.

Running a list command strings with subprocess popen and getting the output

When you create a new process, you don't pass it a list of commands to run; rather, you pass it a single command -- either as a string (with shell=True) or as a list of args (with shell=False).

import subprocess

cmds = ['sleep 1', 'uptime', 'ls -l /']

for cmd in cmds:
    stdout = subprocess.check_output(cmd, shell=True)
    print('\n# {}'.format(cmd))
    print(stdout)

If you just want to collect stdout, subprocess.check_output() might be simpler than Popen() -- but either approach will work, depending on what you need to do with the process.

subprocess.call() arguments ignored when using shell=True w/ list

When shell is True, the first argument is appended to ["/bin/sh", "-c"]. If that argument is a list, the resulting list is

["/bin/sh", "-c", "ls", "-al"]

That is, only ls, not ls -al is used as the argument to the -c option. -al is used as the first argument the shell itself, not ls.

When using shell=True, you generally just want to pass a single string and let the shell split it according the shell's normal word-splitting rules.

# Produces ["/bin/sh", "-c", "ls -al"]
subprocess.call("ls -al", shell=True)

In your case, it doesn't see like you need to use shell=True at all.

---

Related Topics