Passing Double Quote Shell Commands in Python to Subprocess.Popen()

Passing double quote shell commands in python to subprocess.Popen()?

I'd suggest using the list form of invocation rather than the quoted string version:

command = ["ffmpeg", "-i", "concat:1.ts|2.ts", "-vcodec", "copy",
           "-acodec", "copy", "temp.mp4"]
output,error  = subprocess.Popen(
                    command, universal_newlines=True,
                    stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()

This more accurately represents the exact set of parameters that are going to be passed to the end process and eliminates the need to mess around with shell quoting.

That said, if you absolutely want to use the plain string version, just use different quotes (and shell=True):

command = 'ffmpeg -i "concat:1.ts|2.ts" -vcodec copy -acodec copy temp.mp4'
output,error  = subprocess.Popen(
                    command, universal_newlines=True, shell=True,
                    stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()

Passing shell commands in python subprocess.Popen() with two quoted words

1. You don't need to add additional quotes because subprocess will automatically add them when needed.

command = ["iostat", "-d", "7", "7", "-p", "sda", "|", "awk", "!/^Device/", "|", "awk", "!/^Linux/"]
device = subprocess.Popen(command, stdout=subprocess.PIPE)
devicetr = device.stdout.read()  # also here was a typo red -> read

2. As @noufal-ibrahim pointer out | doesn't work this way. You need to pipe your commands:

ps1 = subprocess.Popen(["iostat"], stdout=subprocess.PIPE)
ps2 = subprocess.Popen(["awk",  "!/^ *KB/"], stdin=ps1.stdout, stdout=subprocess.PIPE)  # filter out line with KB and extra space
print(ps2.stdout.read().decode())

^ This code works on my machine, you can adjust it to your use case.

Appendix. And instead of manually creating command list you can use shlex lib (from standard python lib):

>>> import shlex
>>> shlex.split("python foo.py 'multi word arg'")
['python', 'foo.py', 'multi word arg']

Python Subprocess: Unable to Escape Quotes

The shell breaks command strings into lists. The quotes tell the shell to put multiple words into a single list item. Since you are building the list yourself, you add the words as a single item without the quotes.

These two Popen commands are equivalent

Popen("command -flag 'foo foo1'", shell=True)
Popen(["command", "-flag", "foo foo1"])

EDIT

This answer deals with escaping characters in the shell. If you don't use the shell, you don't add any quotes or escapes, just put in the string itself. There are other issues with skipping the shell, like piping commands, running background jobs, using shell variables and etc. These all can be done in python instead of the shell.

Python subprocess.Popen fails with multiple arguments involving double quotes

Your function definition is weird. A more sane design would be to accept a list just like subprocess.Popen and simply pass it on verbatim.

Tangentially, this looks like you should simply use subprocess.run() instead. If you don't particularly care about the output, just run the process and let it write directly to standard output, without passing through Python at all.

subprocess.run(['ls', '-la'])
subprocess.run(['grep', '-r', 'SomeString', '.'])

If you do want the output to be visible to Python, use print(subprocess.check_output(['ls', '-la'], text=True)) or something along those lines.

Subprocess.popen() cannot use quotation marks within arguments on Windows

On Windows, a string is a native API. To avoid unnecessary conversions, pass the command as a string:

#!/usr/bin/env python
from __future__ import print_function
import subprocess

command = 'runme.bat --include="check|check2"'
process = subprocess.Popen(command,
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
    universal_newlines=True, bufsize=1)
for line in iter(process.stdout.readline, ''):
    print(line, end='')

stderr=subprocess.STDOUT merges stderr into stdout. If you set stderr=PIPE then you should read from process.stderr in parallel with reading from process.stdout otherwise your program may deadlock.

Popen() passes the string to CreateProcess() Windows function. If the child process is actually a batch-file; you should probably pass shell=True explicitly to make it clear that the command is interpreted using cmd.exe rules (^, |, etc are meta-characters, for more details read the links in this answer).

If you want to pass the argument using %1 instead of %* so that it includes

the whole --include="check|check2" (not only --include) then you could use additional quotes around the argument as @eryksun suggested in the comments:

command = '"runme.bat" "--include="check^^^|check2""'

Notice: triple ^ to escape | here.

Passing arguments wrapped with double quotes in a subprocess command

You don’t need the double quotes.

The reason you need them with the shell command is that the shell is parsing the command line from a string, and it uses them as an indication that the text run app.package.info -a com.package.name should be placed in a single argument:

#!/usr/bin/env python
import subprocess

c_arg = 'run app.package.info -a com.package.name'
command = ['/usr/bin/drozer', 'console', 'connect', '-c', c_arg]
output = subprocess.check_output(command)
print("Got %r" % (output,))

When you’re using code to start a process, you explicitly specify individual arguments, and no shell parsing is going on, so there’s no need for the extra quotes.

Python Subprocess command with quotes and variables

With subprocess, you're better off passing a list of strings rather than a string to be evaluated by the shell. This way you don't need to worry about balancing your double quotes (and escaping potentially executable values).

The curly braces can be escaped from string formatting by doubling them.

With those two notes in mind, here's what I might do:

committerUser = 'alice'
reviewerUser = 'joe'
branchName = 'testdevbranch'
cmd = ["gitlab",
    "create_merge_request",
    "5",
    f"{committerUser} - New merge request - {reviewerUser}",
    f"{{source_branch: '{branchName}', target_branch: 'dev', assignee_id: 1}}",
    "--json"]
subprocess.Popen(cmd, …)

I'm using Python 3.6's f-strings here, but it could also be done with the str.format() method

"{} - New merge request - {}".format(committerUser, reviewerUser),
"{{source_branch: '{}', target_branch: 'dev', assignee_id: 1}}".format(branchName),

Or explicitly by concatenation, which might be more readable than trying to remember what the double curly braces are for.

committerUser + " - New merge request - " + reviewerUser,
"{source_branch: '" + branchName + "', target_branch: 'dev', assignee_id: 1}",

---

Related Topics