Why Not Just Use 'Shell=True' in Subprocess.Popen in Python

Actual meaning of 'shell=True' in subprocess

The benefit of not calling via the shell is that you are not invoking a 'mystery program.' On POSIX, the environment variable SHELL controls which binary is invoked as the "shell." On Windows, there is no bourne shell descendent, only cmd.exe.

So invoking the shell invokes a program of the user's choosing and is platform-dependent. Generally speaking, avoid invocations via the shell.

Invoking via the shell does allow you to expand environment variables and file globs according to the shell's usual mechanism. On POSIX systems, the shell expands file globs to a list of files. On Windows, a file glob (e.g., "*.*") is not expanded by the shell, anyway (but environment variables on a command line are expanded by cmd.exe).

If you think you want environment variable expansions and file globs, research the ILS attacks of 1992-ish on network services which performed subprogram invocations via the shell. Examples include the various sendmail backdoors involving ILS.

In summary, use shell=False.

how to avoid shell=True in subprocess

Just pass the arguments to check_output() as a list:

subprocess.check_output(["md5", "Downloads/test.txt"], stderr=subprocess.STDOUT)

From the docs:

args is required for all calls and should be a string, or a sequence
of program arguments. Providing a sequence of arguments is generally
preferred, as it allows the module to take care of any required
escaping and quoting of arguments (e.g. to permit spaces in file
names). If passing a single string, either shell must be True (see
below) or else the string must simply name the program to be executed
without specifying any arguments.

Why does this Python subprocess command only work when shell=True on Windows?

I suspect you're calling C:\Windows\System32\convert.exe (NTFS/FAT partition converter) instead of imagemagick.

When shell=True, the path finds the convert.bat or convert.cmd script from imagemagick, but without it, the path can only find the .exe file, which is a completely different program, and you get error 4: invalid parameter.

In that particular case, it doesn't work even with an executable, because the "wrong" convert is located in a system path. shell=False only searches in system paths (python subprocess Popen environment PATH?). So that's bad luck that a program named convert is located in the system path.

Try to explicitly add .bat extension like this:

subprocess.call(['convert.bat', file1, '-resize', '200%', file1])

To know which executables are likely to be run you can type:

where convert

in a command prompt.

In your case (an executable), that could be workarounded by passing the absolute path of the executable you want to run.

Another way would be to copy/rename ImageMagick convert to imconvert. Which program calls itself convert and doesn't expect conflicts anyway ?

Or in that case, it's legitimate to leave shell=True, with a nice comment explaining that Microsoft left a confusing (and seldom used convert program in a system path for us to trip into)

Solutions are not pretty, at least there are some.

Python subprocess.Popen error handling with shell=True/False issues

If you set the shell arg to False, then the args argument becomes a sequence of strings, not a single string: the first args element is the name of the program to run, and subsequent elements are the arguments to that program, e.g. (if I understand your code correctly):

subprocess.Popen \
  (
    [
        "openssl", "aes-128-cbc", "-salt",
        "-in", os.path.join(SrcDIR, var1),
        "-out", os.path.join(DstDIR, "enc." + var1),
        "-k", var2
    ]
  )

This is also better than trying to pass a single command string with shell = True, because this way you don’t have to beware of characters with special meaning to the shell.

Why does subprocess.Popen() with shell=True work differently on Linux vs Windows?

Actually on Windows, it does use cmd.exe when shell=True - it prepends cmd.exe /c (it actually looks up the COMSPEC environment variable but defaults to cmd.exe if not present) to the shell arguments. (On Windows 95/98 it uses the intermediate w9xpopen program to actually launch the command).

So the strange implementation is actually the UNIX one, which does the following (where each space separates a different argument):

/bin/sh -c gcc --version

It looks like the correct implementation (at least on Linux) would be:

/bin/sh -c "gcc --version" gcc --version

Since this would set the command string from the quoted parameters, and pass the other parameters successfully.

From the sh man page section for -c:

Read commands from the command_string operand instead of from the standard input. Special parameter 0 will be set from the command_name operand and the positional parameters ($1, $2, etc.) set from the remaining argument operands.

This patch seems to fairly simply do the trick:

--- subprocess.py.orig  2009-04-19 04:43:42.000000000 +0200
+++ subprocess.py       2009-08-10 13:08:48.000000000 +0200
@@ -990,7 +990,7 @@
                 args = list(args)

             if shell:
-                args = ["/bin/sh", "-c"] + args
+                args = ["/bin/sh", "-c"] + [" ".join(args)] + args

             if executable is None:
                 executable = args[0]

subprocess.Popen(cl, ..., shell=True) does not work like a shell command forwarder

You didn't wait for any of the processes to finish. To make it equivalent to running the shell commands one by one, you need to put a process.wait() call after each Popen so the command finishes before you launch the next process. In this case, aplay depends on pico2wave writing out aplay's input, and depends on rm not removing it before it gets a chance to open it and read the contents.

Without the process.wait(), it's like running (note backgrounding &) this in the shell:

pico2wave --lang=de-DE --wave=/tmp/test.wav "Test" &
aplay /tmp/test.wav &
rm -f /tmp/test.wav &

which introduces all kinds of terrible race conditions.

---

Related Topics