Executing Shell Command in Background from Ruby with Proper Argument Escaping

Executing shell command in background from ruby with proper argument escaping

In Ruby 1.9, try Process.spawn:

# Spawn a new process and run the rake command
pid = Process.spawn({"subject" => params[:subject]},
                    "rake", "send_mails",
                    :out => 'dev/null', :err => 'dev/null')

# Detach the spawned process
Process.detach pid

How to call shell commands from Ruby

This explanation is based on a commented Ruby script from a friend of mine. If you want to improve the script, feel free to update it at the link.

First, note that when Ruby calls out to a shell, it typically calls /bin/sh, not Bash. Some Bash syntax is not supported by /bin/sh on all systems.

Here are ways to execute a shell script:

cmd = "echo 'hi'" # Sample string that can be used

Kernel#` , commonly called backticks – `cmd`
This is like many other languages, including Bash, PHP, and Perl.
Returns the result (i.e. standard output) of the shell command.
Docs: http://ruby-doc.org/core/Kernel.html#method-i-60
```
value = `echo 'hi'`
value = `#{cmd}`
```
Built-in syntax, %x( cmd )
Following the x character is a delimiter, which can be any character.
If the delimiter is one of the characters (, [, {, or <,
the literal consists of the characters up to the matching closing delimiter,
taking account of nested delimiter pairs. For all other delimiters, the
literal comprises the characters up to the next occurrence of the
delimiter character. String interpolation #{ ... } is allowed.
Returns the result (i.e. standard output) of the shell command, just like the backticks.
Docs: https://docs.ruby-lang.org/en/master/syntax/literals_rdoc.html#label-Percent+Strings
```
value = %x( echo 'hi' )
value = %x[ #{cmd} ]
```
Kernel#system
Executes the given command in a subshell.
Returns true if the command was found and run successfully, false otherwise.
Docs: http://ruby-doc.org/core/Kernel.html#method-i-system
```
wasGood = system( "echo 'hi'" )
wasGood = system( cmd )
```
Kernel#exec
Replaces the current process by running the given external command.
Returns none, the current process is replaced and never continues.
Docs: http://ruby-doc.org/core/Kernel.html#method-i-exec
```
exec( "echo 'hi'" )
exec( cmd ) # Note: this will never be reached because of the line above
```

Here's some extra advice:
$?, which is the same as $CHILD_STATUS, accesses the status of the last system executed command if you use the backticks, system() or %x{}.
You can then access the exitstatus and pid properties:

$?.exitstatus

For more reading see:

http://www.elctech.com/blog/i-m-in-ur-commandline-executin-ma-commands
http://blog.jayfields.com/2006/06/ruby-kernel-system-exec-and-x.html
http://tech.natemurray.com/2007/03/ruby-shell-commands.html

Why does ruby’s `system` not `shellescape` the first argument?

Blindly sending a whole command through #shellescape rarely does the right thing. Consider this:

> puts '(pancakes) --house'.shellescape
\(pancakes\)\ --house

Sure, you probably want to escape the parentheses but you almost certainly don't want to escape the space. In order to do The Right Thing and DWIM (Do What I Mean), #system would have to make guesses about which parts should be escaped and which parts shouldn't and it would invariably do the wrong thing half the time.

The solution is to never ever use the single argument form of #system so that you never invoke a shell at all. If you say things like:

system('/bin/ls -l')

then a shell is invoked with that command line and the shell has to parse it before invoking the command. If you intend say:

system('/bin/ls', '-l')

then /bin/ls is invoked directly, no shell is involved so there's no escaping to worry about. Of course, that sometimes leads to silly things like:

system('/bin/ls', '--')
system(['/bin/ls', '--'])

invoke a command without arguments and without a shell but presumably that's not very common and you can fall back on manual escaping when you know that's what you're dealing with.

I tend to pretend that #system doesn't exist and go straight to Open3 when I need to interact with external programs. Open3 is slightly more verbose but has a cleaner and easier to use interface IMO.

Similar arguments apply to using backticks to execute a command and capture its output.

Your second example:

# Unescaped directory as second argument. Success.
> system('touch', '/tmp/(example)/script')
=> true

works because no shell is invoked so the parentheses in the second argument have no special meaning.

Your third example:

# Escaped directory as second argument. Failure.
> system('touch', '/tmp/(example)/script'.shellescape)
touch: /tmp/\(example\)/script: No such file or directory
=> false

fails for the same reason: no shell means no escaping.

Forming sanitary shell commands or system calls in Ruby

It doesn't look like you need a shell for what you're doing. See the documentation for system here: http://ruby-doc.org/core/classes/Kernel.html#M001441

You should use the second form of system. Your example above would become:

system 'usermod', '-p', @options['shadow'], @options['username']

A nicer (IMO) way to write this is:

system *%W(usermod -p #{@options['shadow']} #{@options['username']})

The arguments this way are passed directly into the execve call, so you don't have to worry about sneaky shell tricks.

How can I make a bash script work correctly when I input an URL as an argument

the url http://www.youtube.com/watch?v=g34B-YOaC7c&ob=av2e must be quoted as it contains a & use:

'http://www.youtube.com/watch?v=g34B-YOaC7c&ob=av2e'

Prevent wildecard expansion with system() function call

In the interest of closing the loop, as Slava didn't post the answer... Copied from his comment:

You can create a shared lib with your own system() function and add it by LD_PRELOAD var

I ended up using this method - and it worked perfectly fine.

How do I execute a program or call a system command?

Use the subprocess module in the standard library:

import subprocess
subprocess.run(["ls", "-l"])

The advantage of subprocess.run over os.system is that it is more flexible (you can get the stdout, stderr, the "real" status code, better error handling, etc...).

Even the documentation for os.system recommends using subprocess instead:

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.

On Python 3.4 and earlier, use subprocess.call instead of .run:

subprocess.call(["ls", "-l"])

Running command-line processes in parallel in Ruby

Here's an Example using Resque. Note I've left escaping out for brevity... you should never pass external inputs directly into shell commands.

class RasterizeWebPageJob
  @queue = :screenshots
  def self.perform(url)
    system("/usr/bin/env DISPLAY=:1 phantomjs rasterize.js #{url} ...")
  end
end

10.times { Resque.enqueue(RasterizeWebPageJob, "http://google.com/") }

Provided you're running enough workers (and there are workers available), they'll execute in parallel. The important thing here is that you put separate jobs onto the queue instead of processing multiple screenshots from within the one job.

I'd advise against using Thread.new in a Rails controller. Queues are much easier (and safer) to manage than Threads.

Executing Shell Command in Background from Ruby with Proper Argument Escaping