Line Buffered Cat

Line Buffered Cat

No, but GNU grep with --line-buffered can do this. Just search for something every line has, such as '^'.

Force line-buffering of stdout in a pipeline

Try unbuffer (man page) which is part of the expect package. You may already have it on your system.

In your case you would use it like this:

unbuffer ./a | tee output.txt

The -p option is for pipeline mode where unbuffer reads from stdin and passes it to the command in the rest of the arguments.

What do fully buffered, line buffered and unbuffered mean in C?

Online C11 standard, 7.21.3/3:

When a stream is unbuffered, characters are intended to appear from the source or at the destination as soon as possible. Otherwise characters may be accumulated and transmitted to or from the host environment as a block. When a stream is fully buffered, characters are intended to be transmitted to or from the host environment as a block when a buffer is filled. When a stream is line buffered, characters are intended to be transmitted to or from the host environment as a block when a new-line character is encountered. Furthermore, characters are intended to be transmitted as a block to the host environment when a buffer is filled, when input is requested on an unbuffered stream, or when input is requested on a line buffered stream that requires the transmission of characters from the host environment. Support for these characteristics is implementation-defined, and may be affected via the setbuf and setvbuf functions.

7.21.3/7:

At program startup, three text streams are predefined and need not be opened explicitly — standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device.

Piping the output to cat

That's because printf's output will get buffered by the libc, write's output will not get buffered. It's a direct, unbuffered operation on a file (stdout)

Read this:

if stdout is a terminal then buffering is automatically set to line buffered, else it is set to buffered

So, you are actually piping to cat - that's why buffering is enabled (try without cat to see)

To turn off the buffering issue the stdbuf command:

stdbuf -o0 ./a.out  | cat 

By the way, that's a really good question to ask for someone who isn't an every day C hacker! Simple and descriptive!

Turn off buffering in pipe

You can use the unbuffer command (which comes as part of the expect package), e.g.

unbuffer long_running_command | print_progress

unbuffer connects to long_running_command via a pseudoterminal (pty), which makes the system treat it as an interactive process, therefore not using the 4-kiB buffering in the pipeline that is the likely cause of the delay.

For longer pipelines, you may have to unbuffer each command (except the final one), e.g.

unbuffer x | unbuffer -p y | z

pipeline command ./a.out | cat, printf(line\n) in a.out is not excuted and no output

An output stream can be in one of three different modes, unbuffered, line buffered, or fully buffered. In unbuffered mode, output is written immediately. In line buffered mode, output is written first to an internal stream buffer until the buffer is full or a newline is written, and then the buffer is flushed to the output. In fully buffered mode, the output is written first to an internal stream buffer until the buffer is full, and then the buffer is flushed to the output. (Some implementations may also flush the output at other times, such as when reading input from an interactive device.)

Input streams can also be in the same three modes, and this determines when read input is made available to the caller of functions that read streams.

The implementation initializes the mode of the standard input (stdin), standard output (stdout), and standard error output (stderr) streams before the main function is called (or at least before any access to the streams). Under certain circumstances, the implementation is allowed to initialize the standard input or standard output to fully buffered mode. The standard input and output streams are initialized to fully buffered mode if and only if the implementation can determine that they are not linked to an interactive device (such as a terminal). (The standard error output stream is never initialized to fully buffered mode.)

Typically, the C runtime library of a POSIX system will call isatty on the underlying file descriptors for the standard input and output streams and set the streams to fully buffered mode if isatty returns 0. This occurs before the main function is called.

When you run "./a.out" with output going to the terminal, the C runtime library determines that output is going to an interactive device and does not set stdout to fully buffered mode. It will be set to one of the other modes, typically line buffered mode. However, when you run "/a.out" with output going to a pipe, the C runtime library determines that output is not going to an interactive device and does set stdout to fully buffered mode. This is why the output is not written to the pipe immediately.

There are two ways to solve your problem. The first is to change the standard output stream to line buffered mode or unbuffered mode before the first call to printf:

setvbuf(stdout, NULL, _IOLBF, 0); // set standard output to line buffered mode

setvbuf(stdout, NULL, _IONBF, 0); // set standard output to unbuffered mode

The other way is to to flush the standard output on demand:

fflush(stdout); // write buffered standard output contents

How can I wrap a command with a line-buffered prompt?

First of all, I was wrong about rlwrap, you can use a prompt with it:

rlwrap -S "> " grep hi

And it works pretty well. But it leaves artifacts of the prompt if you have something typed when something is printed by the process.

Then I found socat, and it can do essentially the same thing as the above (among other things), but it doesn't leave those artifacts (by blocking stdout while you type until you press enter and the line is clear again):

socat READLINE,prompt="[IN] " EXEC:"stdbuf -oL grep hi"
[IN] hello
[IN] this is another example
this is another example
[IN] it sure is, i'm glad you know

Then I can just use sed to add a prompt to the output:

socat READLINE,prompt="[IN] " SYSTEM:"stdbuf -oL grep hi | stdbuf -oL sed \'s/^/[OUT] /\'"
[IN] hello
[IN] this is another example
[OUT] this is another example
[IN] it sure is, i'm glad you know

Line buffering in a bash coproc

One way to force line-buffering is to use the stdbuf coreutils tool, if available:

stdbuf allows one to modify the buffering operations of the three standard I/O streams associated with a program.

For the coproc case:

$ coproc stdbuf -oL paste
[3] 42751
$ echo 'Hello world!' >&${COPROC[1]}
$ read -ru ${COPROC[0]} line; echo $line
Hello world!
$ kill $COPROC_PID
[3]+  Terminated              coproc COPROC stdbuf -oL paste
$ 

For the simple pipe of paste to cat case:

$ stdbuf -oL paste | cat
Hello World!<RETURN>
Hello World!^D
$ 

unix command 'tail' lost option '--line-buffered'

As far as can be told by simple googling, tail doesn't appear to have a --line-buffered option, grep does. --line-buffered is useful to force line buffering even when writing to a non-TTY, a typical idiom being:

tail -f FILE | grep --line-buffered REGEXP > output

Here the point of --line-buffered is to prevent grep from buffering output in 8K chunks and forcing the matched lines to immediately appear in the output file.

tail -f is unbuffered regardless of output type, so it doesn't need a --line-buffered option equivalent to the one in grep. This can be verified by running tail -f somefile | cat and appending a line to the file from another shell. One observes that, despite its standard output being a pipe, tail immediately flushes the newly arrived line.

---

Related Topics