Why Does Runtime.Exec(String) Work for Some But Not All Commands

Why does Runtime.exec(String) work for some but not all commands?

Why do some commands fail?

This happens because the command passed to Runtime.exec(String) is not executed in a shell. The shell performs a lot of common support services for programs, and when the shell is not around to do them, the command will fail.

When do commands fail?

A command will fail whenever it depends on a shell features. The shell does a lot of common, useful things we don't normally think about:

1. The shell splits correctly on quotes and spaces

   This makes sure the filename in "My File.txt" remains a single argument.

   Runtime.exec(String) naively splits on spaces and would pass this as two separate filenames. This obviously fails.

2. The shell expands globs/wildcards

   When you run ls *.doc, the shell rewrites it into ls letter.doc notes.doc.

   Runtime.exec(String) doesn't, it just passes them as arguments.

   ls has no idea what * is, so the command fails.

3. The shell manages pipes and redirections.

   When you run ls mydir > output.txt, the shell opens "output.txt" for command output and removes it from the command line, giving ls mydir.

   Runtime.exec(String) doesn't. It just passes them as arguments.

   ls has no idea what > means, so the command fails.

4. The shell expands variables and commands

   When you run ls "$HOME" or ls "$(pwd)", the shell rewrites it into ls /home/myuser.

   Runtime.exec(String) doesn't, it just passes them as arguments.

   ls has no idea what $ means, so the command fails.

What can you do instead?

There are two ways to execute arbitrarily complex commands:

Simple and sloppy: delegate to a shell.

You can just use Runtime.exec(String[]) (note the array parameter) and pass your command directly to a shell that can do all the heavy lifting:

// Simple, sloppy fix. May have security and robustness implications
String myFile = "some filename.txt";
String myCommand = "cp -R '" + myFile + "' $HOME 2> errorlog";
Runtime.getRuntime().exec(new String[] { "bash", "-c", myCommand });

Secure and robust: take on the responsibilities of the shell.

This is not a fix that can be mechanically applied, but requires an understanding the Unix execution model, what shells do, and how you can do the same. However, you can get a solid, secure and robust solution by taking the shell out of the picture. This is facilitated by ProcessBuilder.

The command from the previous example that requires someone to handle 1. quotes, 2. variables, and 3. redirections, can be written as:

String myFile = "some filename.txt";
ProcessBuilder builder = new ProcessBuilder(
    "cp", "-R", myFile,        // We handle word splitting
       System.getenv("HOME")); // We handle variables
builder.redirectError(         // We set up redirections
    ProcessBuilder.Redirect.to(new File("errorlog")));
builder.start();

Java Runtime.exec() works for some command but not others

Because of shell parsing.

These are all concepts that the OS just does not have:

• The concept of 'every space separates one argument from another (and the command from the list of arguments'). The concept that a single string can possibly run anything, in fact; at the OS level that's just not what the 'run this thing' API looks like. When you type commands on the command prompt, your shell app is 'interpreting' these strings into a command execution request.
• The concept of figuring out that bash means /bin/bash, i.e. $PATH resolution.
• The concept that *.txt is supposed to refer to all files that ends in .txt.
• The concept that $FOO should be replaced with the value of the environment variable 'FOO'
• The concept that ; separates 2 commands, and it's supposed to run both.
• The concept that single and double quotes escape things. "Escape things" implies that things can cause interpretation to happen. The OS interprets nothing, therefore there's nothing to escape. Obvious, then, that the OS doesn't know what ' is, or ".
• That >foo means: Please set the standard output of the spawned process such that it sends it all to file 'foo'.
• In windows shells, that @ in front of the command means 'do not echo the command itself'. This and many other things are shellisms: /bin/bash does that, maybe /bin/zsh does something else. Windows's built in shell thing definitely is quite different from bash!

Instead, an OS simply wants you to provide it a full path to an executable along with a list of strings, and pick targets for standard in, standard out, and standard err. It does no processing on any of that, just starts the process you named, and passes it the strings verbatim.

You're sort of half there, as you already figured out that e.g. ls >foo cannot work if you execute it on its own, but it can work if you tell bash to execute it. As ALL of that stuff in the above list? Your shell does that.

It gets more complicated: Turning *.txt into foo.txt bar.txt is a task of bash and friends, e.g. if you attempted to run: ls '*.txt' it does not work. But on windows, it's not the shell's job; the shell just passes it verbatim to dir, and it is the command's job to undo it. What a mess, right? Executing things is hard!

So, what's wrong here? Two things:

• Space splitting isn't working out.
• Quote application isn't being done.

When you write:

bash -c 'ls >foo'

in a bash shell, bash has to first split this up, into a command, and a list of arguments. Bash does so as follows:

• Command: bash
• arg1: -c
• arg2: ls >foo

It knows that ls >foo isn't 2 arguments because, effectively, "space" is the bash operator for: "... and here is the next argument", and with quotes (either single or double), the space becomes a literal space instead of the '... move to next argument ...' operator.

In your code, you ask bash to run java, and then java to run bash. So, bash first does:

• cmd: java
• arg1: bash -c 'ls >foo'

With the same logic at work. Your java app then takes that entire string (that's args[0]: "bash -c 'ls >foo'"), and you then feed it to a method you should never use: Runtime.exec(). Always use ProcessBuilder, and always use the list-based form of it.

Given that you're using the bad method, you're now asking java to do this splitting thing. After all, if you just tell the OS verbatim, please run "bash -c 'ls >foo'", the OS dutifully reports: "I'm sorry, but file ./bash -c ;ls >foo' does not exist", because it does not processing at all". This is unwieldy so java's exec() method is a disaster you should never use: Because people are confused about this, it tries to do some extremely basic processing, except every OS and shell is different, java does not know this, so it does a really bad job at it.

Hence, do not use it.

In this case, java doesn't realize that those quotes mean it shouldn't split, so java tells the OS:

Please run:

• cmd: /bin/bash (java DOES do path lookup; but you should avoid this, do not use relative path names, you should always write them out in full)
• arg1: -c
• arg2: 'ls
• arg3: >foo'

and now you understand why this is just going completely wrong.

Instead, you want java to tell the OS:

• cmd: /bin/bash
• arg1: -c
• arg2: ls >foo

Note: ls >foo needs to be one argument, and NOTHING in the argument should be containing quotes, anywhere. The reason you write:

/bin/bash -c 'ls >foo'

In bash, is because you [A] want bash not to treat that space between ls and >foo as an arg separator (you want ls >foo to travel to /bin/bash as one argument), and [B] that you want >foo to just be sent to the bash you're spawning and not to be treated as 'please redirect the output to file foo' at the current shell level.

Runtime.exec isn't a shell, so the quotes stuff? Runtime.exec has no idea.

This means more generally your plan of "I shall write an app where the entire argument you pass to it is just run" is oversimplified and can never work unless you write an entire quotes and escaper analyser for it.

An easy way out is to take the input, write it out to a shell script on disk, set the exec flag on it, and always run /bin/bash -c /path/to/script-you-just-wrote, sidestepping any need to attempt to parse anything in java: Let bash do it.

The ONE weird bizarro thing I cannot explain, is that literally passing 'ls' to /bin/bash -c, with quotes intact, DOES work and runs ls as normal, but 'ls *' does not, perhaps because now bash thinks you want executable file /bin/ls * which obviously does not exist (a star cannot be in a file name, or at least, that's not the ls executable, and it's not an alias for the ls built-in). At any rate, you want to pass ls without the quotes.

Let's try it!

import java.util.*;
import java.io.*;

public class runtime {
    public static void main(String args[]) throws Exception{
        ProcessBuilder pb = new ProcessBuilder();
        pb.command("/bin/bash", "-c", "echo 1234");
//        pb.command("/bin/bash", "-c", "'echo 1234'");
        Process p = pb.start();
        OutputStream os = p.getOutputStream();
        InputStream in = p.getInputStream();
        DataInputStream dis = new DataInputStream(in);
        String disr = dis.readLine();
        while ( disr != null ) {
                System.out.println("Out: " + disr);
                disr = dis.readLine();
        }
        int errCode = p.waitFor();
        System.out.println("Process exit code: " + errCode);
    }
}

The above works fine. Replace the .command line with the commented out variant and you'll notice it does not work at all, and you get an error. On my mac I get a '127' error; perhaps this is bash reporting back: I could not find the command you were attempting to execute. 0 is what you're looking for when you invoke waitFor: That's the code for 'no errors'.

Does Runtime.exec() work only for certain commands?

The program you're trying to execute is called tree.com.

It is the command-line interpreter cmd.exe that uses the PATHEXT environment variable to search the path for programs with various extensions. Java's API doesn't.

So you have 2 choices:

• Add the extension:

  Process cmd = Runtime.getRuntime().exec("tree.com \"path\" /f /a");
  

• Run it using cmd.exe:

  Process cmd = Runtime.getRuntime().exec("cmd.exe /c tree \"path\" /f /a");
  

The .exe extension is optional, e.g. "cmd /c tree \"path\" /f /a" works too, but any other extension (e.g. .com) is required, and scripts (.bat, .cmd) must be run with cmd.exe.

Command works in terminal, but not with Runtime.exec

This is the most common mistake of all times when it comes to a Process.

A process is not a shell interpreter. As such, any special shell "keywords" will not be interpreted.

If you try and exec cmd1 && cmd2, what happens is that the arguments of the process are literally cmd1, &&, cmd2. Don't do that.

What is more, don't use Runtime.exec(). Use a ProcessBuilder instead. Sample code:

final Process p = new ProcessBuilder("cmd1", "arg1", "arg2").start();
final int retval = p.waitFor();

See the javadoc for ProcessBuilder, it has a lot of niceties.

Oh, and if you use Java 7, don't even bother using external commands. Java 7 has Files.copy().

And also, man execve.

Runtime.getRuntime().exec() doesn't execute some commands

The cd command is a shell built-in command. There is no shell when you run a command via exec(...). Indeed, if you try to find a cd command in any of your system's bin directories, you won't find one ... because it is impossible to implement as a regular command.

If you are trying to use cd to change the current directory for the JVM itself, that won't work because a command can only change the current directory of itself and (after that) commands that it launches itself. It can't change its parent processes current directory.

If you are trying to use cd to change the current directory for subsequent commands, that won't work either. The context in which you set the current directory ends when the command finishes.

In fact, the right way to change the directory for a command run using exec is to set it via the ProcessBuilder API itself.

---

How to make my program run all kinds of terminal commands?

You can't. Some of the "terminal commands" only make sense as shell commands, and that means you need a shell.

I suppose, you could consider emulating the required behaviour in your Java code. That would work for cd ... but other commands are likely to be more difficult to cope with.

(For what it is worth, it is possible to implement a POSIX compatible shell in Java. It is just a LOT of work.)

Runtime.exec() issues with java when attempting to execute scripts, but certain commands work

Ok I ended up having success by using

process p = Runtime.getRuntime().exec("./pyshellthing.sh");
p.waitFor();

The shell script simply executes the python script.

Java Runtime.exec escaped arguments in string

As I said, I have no control over this... I need to do it in one string :(

There is no direct solution given that constraint. Period.

It is a plain fact that exec(String) does not understand any form of escaping or quoting. It splits the string into a command name and multiple arguments using whitespace characters as the argument separator. The behavior is hard-wired ... and documented.

---

The possible solutions are:

• Do the splitting yourself, and use exec(String[]).

• Get a shell to do the splitting; e.g.

  String cmd = "wget -qO- --post-data \"<?xml version=\\"1.0\\" ...."
  Runtime.getRuntime().exec("/bin/sh", "-c", cmd);
  

  Note that we are using exec(String[]) here too.

• Generate and run a shell script on the fly:

  1. Write the following script to a temporary file (say "/tmp/abc.sh")

     #!/bin/sh
     wget -qO- --post-data \
       "<?xml version=\"1.0\" encoding=\"UTF-8\"?><Devices><Device><FLOWCTRL>2</FLOWCTRL></Device></Devices>" \
       -- http://192.168.3.33/data/changes.xml
     

  2. Make the script executable

  3. Run it:

      Runtime.getRuntime().exec("/tmp/abc.sh");
     

java getRuntime().exec() does not work for running basic cmd commands

I solved this issue by using ProcessBuilder. I still don't know why the earlier code didn't work for all the commands. But by using ProcessBuilder, I was able to perform any cmdquery.

Here's the code for reference:

String command_ping = "ping " + host_name;

            try {

                ProcessBuilder builder = new ProcessBuilder("cmd.exe", "/c", command_ping);
                builder.redirectErrorStream(true);
                Process p = builder.start();

                BufferedReader r = new BufferedReader(new InputStreamReader(p.getInputStream()));
                StringBuffer buffer = new StringBuffer();
                String line = "";
                while (true) 
                {

                    buffer.append(line).append("\n");
                    line = r.readLine();
                    if (line == null) { break; }
                }
                message_ping= buffer.toString();
                p.waitFor();
                r.close();

            } 

            catch (IOException e) 
            {
                e.printStackTrace();
            } 

            catch (InterruptedException e) 
            {
                e.printStackTrace();
            }

---

Related Topics