Unbuffered I/O in Linux

UNIX buffered vs unbuffered I/O

Unbuffered I/O simply means that don't use any buffer while reading or writing.Generally when we use System calls like read() and write() they read and write char by char and can cause huge performance degradation . So for huge date generally high level reads/writes or simply buffered I/O are preferred .Buffered simply means that we are not dealing with single char but a block of chars, that is why sometimes it also known as block I/O.Generally in Unix when we use high level read/write functions they fetch/store the data of a given block size and place them in buffer cache and from this buffer cache these I/O functions get the desired amount of data.

Difference between buffered io and unbuffered io

There are multiple layers of buffering. If you call write, no application layer buffering will occur. If you look at the file from another process you will see the data, but that does not mean they have been committed to disk, because there is a layer of buffering happening in the kernel. Since the kernel is handling the access from the other process, it is reporting the data in the buffer to that other process. In other words, from the perspective of all user-space applications the data has been written to the file, but it has not actually hit the disk.

About the unbuffered I/O in UNIX Systems

There are several kinds of buffering going on. The input to the program is buffered by the pseudoterminal device's Line-buffering discipline. On the output side, there is a file-system cache (a buffer in the OS for the whole file), and extra buffering in the C program when printing to a FILE * type. But read and write bypass the FILE * buffering and move data more or less directly to/from the file-system cache.

So it appears that your stdout buffer is being flushed automatically when all output is going to the terminal, but not when redirected to a file. So I'd recommend adding a call to

fflush(stdout);

after the printf call. This should explicitly flush the buffer (and enforce the ordering of the output that you want).

The important thing to be aware of is when you're using FILE *s which are a C-level structure manipulated by library functions (like fopen), and when you're using the raw file descriptor (which is just an integer, but refers to the underlying operating-system file). The FILE datatype is a wrapper around this lower level Unix implementation detail. The FILE functions implement an additional layer of buffering so the lower level can operate on larger blocks, and you can efficiently perform byte-at-a-type processing without doing lots and lots I/O handshakes.

Buffered vs unbuffered IO

You want unbuffered output whenever you want to ensure that the output has been written before continuing. One example is standard error under a C runtime library - this is usually unbuffered by default. Since errors are (hopefully) infrequent, you want to know about them immediately. On the other hand, standard output is buffered simply because it's assumed there will be far more data going through it.

Another example is a logging library. If your log messages are held within buffers in your process, and your process dumps core, there a very good chance that output will never be written.

In addition, it's not just system calls that are minimized but disk I/O as well. Let's say a program reads a file one byte at a time. With unbuffered input, you will go out to the (relatively very slow) disk for every byte even though it probably has to read in a whole block anyway (the disk hardware itself may have buffers but you're still going out to the disk controller which is going to be slower than in-memory access).

By buffering, the whole block is read in to the buffer at once then the individual bytes are delivered to you from the (in-memory, incredibly fast) buffer area.

Keep in mind that buffering can take many forms, such as in the following example:

+-------------------+-------------------+
| Process A         | Process B         |
+-------------------+-------------------+
| C runtime library | C runtime library | C RTL buffers
+-------------------+-------------------+
|               OS caches               | Operating system buffers
+---------------------------------------+
|      Disk controller hardware cache   | Disk hardware buffers
+---------------------------------------+
|                   Disk                |
+---------------------------------------+

Why do unbuffered read()/write() operations use buffer cache?

Basically the term "buffering" here means "a place where data is stored when going to/from the kernel", i.e. to avoid doing one system call for each I/O call, the buffered functions use a buffer between.

What the kernel does with the data is not something the standard library can do much about.

It would be possible to do a 1:1 mapping of read/write calls at the standard library's level (i.e. fread() and friends) to read()/write() calls on the underlying file descriptor; the term buffering is telling you that is not what you can expect.

What do fully buffered, line buffered and unbuffered mean in C?

Online C11 standard, 7.21.3/3:

When a stream is unbuffered, characters are intended to appear from the source or at the destination as soon as possible. Otherwise characters may be accumulated and transmitted to or from the host environment as a block. When a stream is fully buffered, characters are intended to be transmitted to or from the host environment as a block when a buffer is filled. When a stream is line buffered, characters are intended to be transmitted to or from the host environment as a block when a new-line character is encountered. Furthermore, characters are intended to be transmitted as a block to the host environment when a buffer is filled, when input is requested on an unbuffered stream, or when input is requested on a line buffered stream that requires the transmission of characters from the host environment. Support for these characteristics is implementation-defined, and may be affected via the setbuf and setvbuf functions.

7.21.3/7:

At program startup, three text streams are predefined and need not be opened explicitly — standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device.

Is there an un-buffered I/O in Windows system?

Look at CreateFile with the FILE_FLAG_NO_BUFFERING option