Processbuilder: Forwarding Stdout and Stderr of Started Processes Without Blocking the Main Thread

ProcessBuilder: Forwarding stdout and stderr of started processes without blocking the main thread

For Java 7 and later, see Evgeniy Dorofeev's answer.

For Java 6 and earlier, create and use a StreamGobbler:

StreamGobbler errorGobbler = 
  new StreamGobbler(p.getErrorStream(), "ERROR");

// any output?
StreamGobbler outputGobbler = 
  new StreamGobbler(p.getInputStream(), "OUTPUT");

// start gobblers
outputGobbler.start();
errorGobbler.start();

...

private class StreamGobbler extends Thread {
    InputStream is;
    String type;

    private StreamGobbler(InputStream is, String type) {
        this.is = is;
        this.type = type;
    }

    @Override
    public void run() {
        try {
            InputStreamReader isr = new InputStreamReader(is);
            BufferedReader br = new BufferedReader(isr);
            String line = null;
            while ((line = br.readLine()) != null)
                System.out.println(type + "> " + line);
        }
        catch (IOException ioe) {
            ioe.printStackTrace();
        }
    }
}

Java Process: read stdout and stderr of a subprocess in a single thread

No, you can't just create a SelectableChannel from the InputStream returned by a Process, this is a Java api limitation AFAICT.

Read an external process's interleaved stdout and stderr exactly as they are written

The comments by John Kugelman pointed my in the right direction. Thank you a lot!

The buffering doesn't happen on the Java side but is the result of piping stdout and stderr to another program. In this case the JVM.

it becomes full buffered. stderr, on the other hand, stays line buffered even when piped.

So we want to change the buffer behavior of the external process call.

I took the script route described in https://unix.stackexchange.com/a/61833/274368.
My code now looks something like this:

String command = "script -q -c \"my-command\" /dev/null";
ProcessBuilder processBuilder = fs.runInShell(command, args.toArray(new String[args.size()]));
processBuilder.directory(workingDirectory);
processBuilder.redirectErrorStream(true);
Process process = processBuilder.start();
try (BufferedReader input = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
    String message = input.lines().collect(Collectors.joining(System.lineSeparator()));
    // ...
}

Where message contains all lines from stderr and stdout in the correct order.

ProcessBuilder hangs

You wait for the program to end before you being reading the piped output, but a pipe only has a limited buffer, so when the buffer is full, the program will wait for you consume the buffered output, but you're waiting on the program to end. Deadlock!

Always consume output before calling waitFor().

---

UPDATE

Recommend you change the code as follows:

val process = ProcessBuilder(command)
    .redirectErrorStream(true)
    .start()
val stdOut = processResult.inputStream.bufferedReader().readText()
if (process.waitFor(timeoutAmount, timeoutUnit)) {
    val exitCode = processResult.exitValue()
    return AppResult(exitCode, stdOut, "")
}
// timeout: decide what to do here, since command hasn't terminated yet

There is no need to specify Redirect.PIPE, since that's the default. If you don't join stderr and stdout like shown here, you'd need to create threads to consume them individually, since they both have the buffer full issue, so you can't just read one of them first.

Running another program but leaving stdout/stderr alone

You can use redirectOutput for stdout, and the similar call redirectError for stderr:

builder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
builder.redirectError(ProcessBuilder.Redirect.INHERIT);

As @Abra points out, the inheritIO() method is a convenience for these two calls as well as the stdin equivalent.

ProcessBuilder redirecting output

Shell redirection operators are unknown to ProcessBuilder. Put your command in a shell script and execute it, as shown here. Alternatively, use bash -c, as shown here.

Process started with ProcessBuilder stop supplying data

I found the answer as I was typing. Apparently, you MUST take care of both STDERR and STDOUT streams somehow, they don't just disappear. If you only read one of them and ignore the other, eventually the ignored one's buffer will get too big and block until it gets emptied.

so adding the following code solved it

new Thread( () -> {
     try (InputStream es = csvBuilder.getErrorStream() ){
            IOUtils.skip(es, Long.MAX_VALUE);
     } catch (IOException e) {
            throw new RuntimeException(e);
     }
}).start();

---

Related Topics