How to Escape Os.System() Calls

How to escape os.system() calls?

This is what I use:

def shellquote(s):
    return "'" + s.replace("'", "'\\''") + "'"

The shell will always accept a quoted filename and remove the surrounding quotes before passing it to the program in question. Notably, this avoids problems with filenames that contain spaces or any other kind of nasty shell metacharacter.

Update: If you are using Python 3.3 or later, use shlex.quote instead of rolling your own.

How do I escape/use single quotes in os.system command with python 2?

After looking through the documentation of imapsync, I found the recommendation to enclose passwords in double quotes within single quotes to avoid common problems.

Since you already start the string with double quotes, you have to escape the double quotes around your password with a backslash \".

There are also two things you could do to make your code even better.
First, you can use .format syntax for string formatting instead of the old % syntax.

Second replace os.system with subprocess.Popen. This allows you to split your command string into a list of all arguments, which looks more clear.

Your new code would look like

import subprocess

args = [
  "imapsync",
  "--host1",
  fromHost,
  "--user1",
  emails,
  "--password1",
  "'\"{}\"'".format(passwords),
  "--host2",
  toHost,
  "--user2",
  emails,
  "--password2",
  "'\"{}\"'".format(passwords),
  "--ssl1",
  "--no-modulesversion",
  "--ssl2"
]

p = subprocess.Popen(args, stdout=subprocess.PIPE)

output = p.communicate()[0]

print(output)

In this example Popen.communicate is used to gather the output of the imapsync command as a string.
The communicate method returns a tuple with the outputs of the subprocess to stdout and stderr streams.

If you also want to read the output to stderr from the subprocess, change the code as following:

p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

output, errors = p.communicate()

print(output)
print(errors)

properly handling shell escaping from Python using os.system

os.system doesn't execute commands in a normal bash environment like you would expect. You can work around it by simply calling bash yourself:

import os
cmd = """/bin/bash -c "cat %s | sort --stable -t $'\t' -k1,1" """ % "test"
os.system(cmd)

But you should be aware that os.system has been marked as deprecated, and will be removed in future versions of python. You can future-proof your code by using subprocess's convenience method call that mimics os.system's behavior:

import subprocess
cmd = """/bin/bash -c "cat %s | sort --stable -t $'\t' -k1,1" """ % "test"
subprocess.call(cmd, shell=True)

There are more ways to make that call with the subprocess module if you are interested:

http://docs.python.org/library/subprocess.html#module-subprocess

Inexplicable shell command un-escaping behavior in Python's os.system and subprocess.check_output on Ubuntu 18.04

While I can agree that the behavior is strange, it is not inexplicable. There is a reason for the behavior, which has nothing to do with Python or subprocess. Exactly the same behavior is seen in a C program, using the system call to the OS (Linux) as with your Python program.

The reason has to do with your shell, though, but not exactly with bash. The reason is rather that when calling os.system() or the subprocess.Popen() family (including subprocess.check_output()) with shell=True. The documentation states that "On POSIX with shell=True, the shell defaults to /bin/sh." Thus, the shell that invokes your echo command is not bash even if that is your default shell and the shell from which you are running your script/starting Python.

Instead, your command is executed by /bin/sh of your system. For a long time, this just pointed to /bin/bash (running in POSIX compliant mode) in almost all Linux versions, however, lately this has changed in some distributions, among them Ubuntu (but not CentOS apparently, since you do not see the same behavior there), which now have /bin/sh point at bin/dash instead:

$ ll /bin/sh
lrwxrwxrwx 1 root root 4 sep 23 12:53 /bin/sh -> dash*

Thus, your script is in fact executed by dash instead of bash. And "for efficiency" (see man dash at the prompt) dash has chosen to internally implement echo instead of using /bin/echo (used by bash). Unfortunately, the dash echo is not as potent as /bin/echo and has a different interpretation of string inputs, namely dash echo does it's escaping of a number of backslash commands, which in effect means it "swallows"
one extra backslash for you.

It is possible to make /bin/echo behave in the same way by specifying the -e option (see man echo) but unfortunately, it is not possible to have the dash builtin echo to not escape backslashes.

Now, this is the reason for what you see. A good way to avoid the problem is to not rely on the system shell invocation. If it is a single command, such as echo it is best to not invoke a shell at all, removing the shell=True flag. Or, if you need some shell specific functionality, control the invocation of the shell yourself. And, a third way, in this particular case, is to explicitly point to /bin/echo while executing, as that ensures that the "standard" echo is used:

#!/usr/bin/env python3
import sys
import subprocess
import shlex

def get_command(n):
    return "echo 'Should be {} backslahes: {}'".format(n, "\\"*n)

print("")
print("Using subprocess.check_output:")
print("")
for n in range(1, 5):

    # Direct invocation:
    cmd = get_command(n)
    sys.stdout.write(subprocess.check_output(shlex.split(cmd)).decode())

    # Controlling invocation shell:
    bash_cmd = ['/bin/bash', '-c'] + [cmd]
    sys.stdout.write(subprocess.check_output(bash_cmd).decode())

    # Using shell=True but point to /bin/echo
    echo_cmd = '/bin/' + cmd
    sys.stdout.write(subprocess.check_output(echo_cmd, shell=True).decode())

Note that when used without shell=True the command should be a list and not a string. This can be shlex.split() as shown.

Of these approaches, the first one (direct echo invocation) is preferred, due to security concerns, if there is any chance of some parameters coming from untrusted sources. In that case, however, shlex.split() should also not be used, as it opens up the same security vulnerabilities.

How to correctly escape system calls from inside R

Strings in R may be enclosed in either single (') or double (") quotes.

If you want to execute a command with both single and double quotes, such as:

perl -e 'print "test\n"'

then it is of little consequence which you choose for your R string - since one pair needs to be escaped either way.

Let's say you choose single quotes:

system('')

Then we need to escape the single quotes in the same way as for the newline character, with the escape character, \:

command <- 'perl -e \'print "test\n"\''
system(command)

It is also possible to encode Unicode characters in this way with \Unnnnnnnn or \unnnn. Alternatively with octal (\nnn), or hex (\xnnn).

Thus:

atSymbol <- '\u0040' # '\x040' '\100'

If the @ in your curl command is causing the problem, encoding it like this should fix it.

---

Related Topics