System("Command") Produces an Error; But It Works When Invoked Directly from Bash Prompt

system( command ) produces an error; but it works when invoked directly from Bash prompt

As mentioned system() creates a new standard shell sh and executes the commands. Since <() is a bash specific feature it can't be interpreted by sh.

You can circumvent this by calling bash explicitly and use the -c option:

system("bash -c \"diff <(cat /etc/passwd) <(ls -l /etc)\"");

or using a raw string literal:

system(R"cmd(bash -c "diff <(cat /etc/passwd) <(ls -l /etc)")cmd");

Here's the relevant part of the system(3) call manual page:

The system() library function uses fork(2) to create a child process
that executes the shell command specified in command using execl(3)
as follows:

 execl("/bin/sh", "sh", "-c", command, (char *) 0);

system() returns after the command has been completed.

Running shell command and capturing the output

In all officially maintained versions of Python, the simplest approach is to use the subprocess.check_output function:

>>> subprocess.check_output(['ls', '-l'])
b'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'

check_output runs a single program that takes only arguments as input.1 It returns the result exactly as printed to stdout. If you need to write input to stdin, skip ahead to the run or Popen sections. If you want to execute complex shell commands, see the note on shell=True at the end of this answer.

The check_output function works in all officially maintained versions of Python. But for more recent versions, a more flexible approach is available.

Modern versions of Python (3.5 or higher): run

If you're using Python 3.5+, and do not need backwards compatibility, the new run function is recommended by the official documentation for most tasks. It provides a very general, high-level API for the subprocess module. To capture the output of a program, pass the subprocess.PIPE flag to the stdout keyword argument. Then access the stdout attribute of the returned CompletedProcess object:

>>> import subprocess
>>> result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE)
>>> result.stdout
b'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'

The return value is a bytes object, so if you want a proper string, you'll need to decode it. Assuming the called process returns a UTF-8-encoded string:

>>> result.stdout.decode('utf-8')
'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'

This can all be compressed to a one-liner if desired:

>>> subprocess.run(['ls', '-l'], stdout=subprocess.PIPE).stdout.decode('utf-8')
'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'

If you want to pass input to the process's stdin, you can pass a bytes object to the input keyword argument:

>>> cmd = ['awk', 'length($0) > 5']
>>> ip = 'foo\nfoofoo\n'.encode('utf-8')
>>> result = subprocess.run(cmd, stdout=subprocess.PIPE, input=ip)
>>> result.stdout.decode('utf-8')
'foofoo\n'

You can capture errors by passing stderr=subprocess.PIPE (capture to result.stderr) or stderr=subprocess.STDOUT (capture to result.stdout along with regular output). If you want run to throw an exception when the process returns a nonzero exit code, you can pass check=True. (Or you can check the returncode attribute of result above.) When security is not a concern, you can also run more complex shell commands by passing shell=True as described at the end of this answer.

Later versions of Python streamline the above further. In Python 3.7+, the above one-liner can be spelled like this:

>>> subprocess.run(['ls', '-l'], capture_output=True, text=True).stdout
'total 0\n-rw-r--r-- 1 memyself staff 0 Mar 14 11:04 files\n'

Using run this way adds just a bit of complexity, compared to the old way of doing things. But now you can do almost anything you need to do with the run function alone.

Older versions of Python (3-3.4): more about check_output

If you are using an older version of Python, or need modest backwards compatibility, you can use the check_output function as briefly described above. It has been available since Python 2.7.

subprocess.check_output(*popenargs, **kwargs)  

It takes takes the same arguments as Popen (see below), and returns a string containing the program's output. The beginning of this answer has a more detailed usage example. In Python 3.5+, check_output is equivalent to executing run with check=True and stdout=PIPE, and returning just the stdout attribute.

You can pass stderr=subprocess.STDOUT to ensure that error messages are included in the returned output. When security is not a concern, you can also run more complex shell commands by passing shell=True as described at the end of this answer.

If you need to pipe from stderr or pass input to the process, check_output won't be up to the task. See the Popen examples below in that case.

Complex applications and legacy versions of Python (2.6 and below): Popen

If you need deep backwards compatibility, or if you need more sophisticated functionality than check_output or run provide, you'll have to work directly with Popen objects, which encapsulate the low-level API for subprocesses.

The Popen constructor accepts either a single command without arguments, or a list containing a command as its first item, followed by any number of arguments, each as a separate item in the list. shlex.split can help parse strings into appropriately formatted lists. Popen objects also accept a host of different arguments for process IO management and low-level configuration.

To send input and capture output, communicate is almost always the preferred method. As in:

output = subprocess.Popen(["mycmd", "myarg"], 
stdout=subprocess.PIPE).communicate()[0]

Or

>>> import subprocess
>>> p = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE,
... stderr=subprocess.PIPE)
>>> out, err = p.communicate()
>>> print out
.
..
foo

If you set stdin=PIPE, communicate also allows you to pass data to the process via stdin:

>>> cmd = ['awk', 'length($0) > 5']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
... stderr=subprocess.PIPE,
... stdin=subprocess.PIPE)
>>> out, err = p.communicate('foo\nfoofoo\n')
>>> print out
foofoo

Note Aaron Hall's answer, which indicates that on some systems, you may need to set stdout, stderr, and stdin all to PIPE (or DEVNULL) to get communicate to work at all.

In some rare cases, you may need complex, real-time output capturing. Vartec's answer suggests a way forward, but methods other than communicate are prone to deadlocks if not used carefully.

As with all the above functions, when security is not a concern, you can run more complex shell commands by passing shell=True.

Notes

1. Running shell commands: the shell=True argument

Normally, each call to run, check_output, or the Popen constructor executes a single program. That means no fancy bash-style pipes. If you want to run complex shell commands, you can pass shell=True, which all three functions support. For example:

>>> subprocess.check_output('cat books/* | wc', shell=True, text=True)
' 1299377 17005208 101299376\n'

However, doing this raises security concerns. If you're doing anything more than light scripting, you might be better off calling each process separately, and passing the output from each as an input to the next, via

run(cmd, [stdout=etc...], input=other_output)

Or

Popen(cmd, [stdout=etc...]).communicate(other_output)

The temptation to directly connect pipes is strong; resist it. Otherwise, you'll likely see deadlocks or have to do hacky things like this.

How do I execute a program or call a system command?

Use the subprocess module in the standard library:

import subprocess
subprocess.run(["ls", "-l"])

The advantage of subprocess.run over os.system is that it is more flexible (you can get the stdout, stderr, the "real" status code, better error handling, etc...).

Even the documentation for os.system recommends using subprocess instead:

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.

On Python 3.4 and earlier, use subprocess.call instead of .run:

subprocess.call(["ls", "-l"])

Hiding console output produced by os.system

To answer the question based on its title in the most generic form:

To suppress all output from os.system(), append >/dev/null 2>&1 to the shell command, which silences both stdout and stderr; e.g.:

import os
os.system('echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null 2>&1')

Note that os.system() by design passes output from the calling process' stdout and stderr streams through to the console (terminal) - your Python code never sees them.

Also, os.system() does not raise an exception if the shell command fails and instead returns an exit code; note that it takes additional work to extract the shell command's true exit code: you need to extract the high byte from the 16-bit value returned, by applying >> 8 (although you can rely on a return value other than 0 implying an error condition).


Given the above limitations of os.system(), it is generally worthwhile to use the functions in the subprocess module instead:

For instance, subprocess.check_output() could be used as follows:

import subprocess
subprocess.check_output('echo 3 | sudo tee /proc/sys/vm/drop_caches', shell=True)

The above will:

  • capture stdout output and return it (with the return value being ignored in the example above)
  • pass stderr output through; passing stderr=subprocess.STDOUT as an additional argument would also capture stderr.
  • raise an error, if the shell command fails.

Note: Python 3.5 introduced subprocess.run(), a more flexible successor to both os.system() and subprocess.check_output() - see https://docs.python.org/3.5/library/subprocess.html#using-the-subprocess-module


Note:

  • The reason that the OP is employing tee in the first place - despite not being interested in stdout output - is that a naïve attempt to use > ... instead would be interpreted before sudo is invoked, and thus fail, because the required privileges to write to /proc/sys/... haven't been granted yet.
  • Whether you're using os.system() or a subprocess function, stdin is not affected by default, so if you're invoking your script from a terminal, you'll get an interactive password prompt when the sudo command is encountered (unless the credentials have been cached).

Python exit commands - why so many and when should each be used?

The functions* quit(), exit(), and sys.exit() function in the same way: they raise the SystemExit exception. So there is no real difference, except that sys.exit() is always available but exit() and quit() are only available if the site module is imported (docs).

The os._exit() function is special, it exits immediately without calling any cleanup functions (it doesn't flush buffers, for example). This is designed for highly specialized use cases... basically, only in the child after an os.fork() call.

Conclusion

  • Use exit() or quit() in the REPL.

  • Use sys.exit() in scripts, or raise SystemExit() if you prefer.

  • Use os._exit() for child processes to exit after a call to os.fork().

All of these can be called without arguments, or you can specify the exit status, e.g., exit(1) or raise SystemExit(1) to exit with status 1. Note that portable programs are limited to exit status codes in the range 0-255, if you raise SystemExit(256) on many systems this will get truncated and your process will actually exit with status 0.

Footnotes

* Actually, quit() and exit() are callable instance objects, but I think it's okay to call them functions.

How do I test if a variable is a number in Bash?

One approach is to use a regular expression, like so:

re='^[0-9]+$'
if ! [[ $yournumber =~ $re ]] ; then
echo "error: Not a number" >&2; exit 1
fi

If the value is not necessarily an integer, consider amending the regex appropriately; for instance:

^[0-9]+([.][0-9]+)?$

...or, to handle numbers with a sign:

^[+-]?[0-9]+([.][0-9]+)?$


Related Topics



Leave a reply



Submit