Unbuffered Bash Output

How to make output of any shell command unbuffered?

AFAIK, you can't do it without ugly hacks. Writing to a pipe (or reading from it) automatically turns on full buffering and there is nothing you can do about it :-(. "Line buffering" (which is what you want) is only used when reading/writing a terminal. The ugly hacks exactly do this: They connect a program to a pseudo-terminal, so that the other tools in the pipe read/write from that terminal in line buffering mode. The whole problem is described here:

• http://www.pixelbeat.org/programming/stdio_buffering/

The page has also some suggestions (the aforementioned "ugly hacks") what to do, i.e. using unbuffer or pulling some tricks with LD_PRELOAD.

bash: force exec'd process to have unbuffered stdout

You might find that the unbuffer script that comes with expect may help.

Turn off buffering in pipe

You can use the unbuffer command (which comes as part of the expect package), e.g.

unbuffer long_running_command | print_progress

unbuffer connects to long_running_command via a pseudoterminal (pty), which makes the system treat it as an interactive process, therefore not using the 4-kiB buffering in the pipeline that is the likely cause of the delay.

For longer pipelines, you may have to unbuffer each command (except the final one), e.g.

unbuffer x | unbuffer -p y | z

Force line-buffering of stdout in a pipeline

Try unbuffer (man page) which is part of the expect package. You may already have it on your system.

In your case you would use it like this:

unbuffer ./a | tee output.txt

The -p option is for pipeline mode where unbuffer reads from stdin and passes it to the command in the rest of the arguments.

Force flushing of output to a file while bash script is still running

bash itself will never actually write any output to your log file. Instead, the commands it invokes as part of the script will each individually write output and flush whenever they feel like it. So your question is really how to force the commands within the bash script to flush, and that depends on what they are.

How do I intercept the unbuffered output of a Proc::Async in Raku?

Proc::Async itself isn't performing buffering on the received data. However, spawned processes may do their own depending on what they are outputting to, and that's what is being observed here.

Many programs make decisions about their output buffering (among other things, such as whether to emit color codes) based on whether the output handle is attached to a TTY (a terminal). The assumption is that a TTY means a human is going to be watching the output, and thus latency is preferable to throughput, so buffering is disabled (or restricted to line buffering). If, on the other hand, the output is going to a pipe or a file, then the assumption is that latency is not so important, and buffering is used to achieve a significant throughput win (a lot less system calls to write data).

When we spawn something with Proc::Async, the standard output of the spawned process is bound to a pipe - which is not a TTY. Thus the invoked program may use this to decide to apply output buffering.

If you're willing to have another dependency, then you can invoke the program via. something that fakes up a TTY, such as unbuffer (part of the expect package, it seems). Here's an example of a program that is suffering from buffering:

my $proc = Proc::Async.new: 'raku', '-e',
    'react whenever Supply.interval(1) { .say }';
react whenever $proc.stdout {
    .print
}

We only see a 0 and then have to wait a long time for more output. Running it via unbuffer:

my $proc = Proc::Async.new: 'unbuffer', 'raku', '-e',
    'react whenever Supply.interval(1) { .say }';
react whenever $proc.stdout {
    .print
}

Means that we see a number output every second.

Could Raku provide a built-in solution to this some day? Yes - by doing the "magic" that unbuffer itself does (I presume allocating a pty - kind of a fake TTY). This isn't trivial - although it is being explored by the libuv developers; at least so far as Rakudo on MoarVM goes, the moment there's a libuv release available offering such a feature, we'll work on exposing it.

---

Related Topics