Actual Meaning of 'Shell=True' in Subprocess

Actual meaning of 'shell=True' in subprocess

The benefit of not calling via the shell is that you are not invoking a 'mystery program.' On POSIX, the environment variable SHELL controls which binary is invoked as the "shell." On Windows, there is no bourne shell descendent, only cmd.exe.

So invoking the shell invokes a program of the user's choosing and is platform-dependent. Generally speaking, avoid invocations via the shell.

Invoking via the shell does allow you to expand environment variables and file globs according to the shell's usual mechanism. On POSIX systems, the shell expands file globs to a list of files. On Windows, a file glob (e.g., "*.*") is not expanded by the shell, anyway (but environment variables on a command line are expanded by cmd.exe).

If you think you want environment variable expansions and file globs, research the ILS attacks of 1992-ish on network services which performed subprogram invocations via the shell. Examples include the various sendmail backdoors involving ILS.

In summary, use shell=False.

Why not just use `shell=True` in subprocess.Popen in Python?

Using shell = True can be a security risk if your input data comes from an untrusted source. E.g. what if the content of your mid variable is "/dev/null; rm -rf /". This does not seem to be the case in your scenario, so I would not worry too much about it.

In your code you write the result of awk directly to the filename in mid. To debug the problem, you might want to use subprocess.check_output and read the result from your awk invocation in your python program.

cmd = """sort -n -r -k5 %s |
      head -n 500|
      awk 'OFS="\t"{{if($2-{1}>0){{print $1,$2-{1},$3+{1},$4,$5}}}}'""".format(summit, top_count)

subprocess.check_call(cmd, shell=True, stdout=file)

What does the 'shell' argument in subprocess mean on Windows?

When you execute an external process, the command you want may look something like "foo arg1 arg2 arg3". If "foo" is an executable, that is what gets executed and given the arguments.

However, often it is the case that "foo" is actually a script of some sort, or maybe a command that is built-in to the shell and not an actual executable file on disk. In this case the system can't execute "foo" directly because, strictly speaking, these sorts of things aren't executable. They need some sort of "shell" to execute them. On *nix systems this shell is typically (but not necessarily) /bin/sh. On windows it will typically be cmd.exe (or whatever is stored in the COMSPEC environment variable).

This parameter lets you define what shell you wish to use to execute your command, for the relatively rare case when you don't want the default.

Why does shell=True eat my subprocess.Popen stdout?

When you pass shell=True, Popen expects a single string argument, not a list. So when you do this:

p1 = Popen(['echo','hello'], stdout=PIPE, shell=True)

What happens is this:

execve("/bin/sh", ["/bin/sh", "-c", "echo", "hello"], ...)

That is, it calls sh -c "echo", and hello is effectively ignored (technically it becomes a positional argument to the shell). So the shell runs echo, which prints \n, which is why you see that in your output.

If you use shell=True, you need to do this:

p1 = Popen('echo hello', stdout=PIPE, shell=True)

---

Related Topics