on-the-fly output redirection, seeing the file redirection output while the program is still running
From the stdout
manual page:
The stream stderr is unbuffered.
The stream stdout is line-buffered
when it points to a terminal.
Partial lines will not appear until
fflush(3) or exit(3) is called, or
a new‐line is printed.
Bottom line: Unless the output is a terminal, your program will have its standard output in fully buffered mode by default. This essentially means that it will output data in large-ish blocks, rather than line-by-line, let alone character-by-character.
Ways to work around this:
Fix your program: If you need real-time output, you need to fix your program. In C you can use
fflush(stdout)
after each output statement, orsetvbuf()
to change the buffering mode of the standard output. For Python there issys.stdout.flush()
of even some of the suggestions here.Use a utility that can record from a PTY, rather than outright stdout redirections. GNU Screen can do this for you:
screen -d -m -L python test.py
would be a start. This will log the output of your program to a file called
screenlog.0
(or similar) in your current directory with a default delay of 10 seconds, and you can usescreen
to connect to the session where your command is running to provide input or terminate it. The delay and the name of the logfile can be changed in a configuration file or manually once you connect to the background session.
EDIT:
On most Linux system there is a third workaround: You can use the LD_PRELOAD
variable and a preloaded library to override select functions of the C library and use them to set the stdout
buffering mode when those functions are called by your program. This method may work, but it has a number of disadvantages:
It won't work at all on static executables
It's fragile and rather ugly.
It won't work at all with SUID executables - the dynamic loader will refuse to read the
LD_PRELOAD
variable when loading such executables for security reasons.It's fragile and rather ugly.
It requires that you find and override a library function that is called by your program after it initially sets the
stdout
buffering mode and preferably before any output.getenv()
is a good choice for many programs, but not all. You may have to override common I/O functions such asprintf()
orfwrite()
- if push comes to shove you may just have to override all functions that control the buffering mode and introduce a special condition forstdout
.It's fragile and rather ugly.
It's hard to ensure that there are no unwelcome side-effects. To do this right you'd have to ensure that only
stdout
is affected and that your overrides will not crash the rest of the program if e.g.stdout
is closed.Did I mention that it's fragile and rather ugly?
That said, the process it relatively simple. You put in a C file, e.g. linebufferedstdout.c
the replacement functions:
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>
char *getenv(const char *s) {
static char *(*getenv_real)(const char *s) = NULL;
if (getenv_real == NULL) {
getenv_real = dlsym(RTLD_NEXT, "getenv");
setlinebuf(stdout);
}
return getenv_real(s);
}
Then you compile that file as a shared object:
gcc -O2 -o linebufferedstdout.so -fpic -shared linebufferedstdout.c -ldl -lc
Then you set the LD_PRELOAD
variable to load it along with your program:
$ LD_PRELOAD=./linebufferedstdout.so python test.py | tee -a test.out
0
1000
2000
3000
4000
If you are lucky, your problem will be solved with no unfortunate side-effects.
You can set the LD_PRELOAD
library in the shell, if necessary, or even specify that library system-wide (definitely NOT recommended) in /etc/ld.so.preload
.
Why does program output redirection make the output of its sub-processes to be out of order?
A terminal generally uses line-buffering, while a pipe would use block buffering.
This means that your printf
call, which includes a newline, will fill the line buffer, triggering a flush. When redirecting, no flush takes place until the program completes.
echo
on the other hand, always flushes the buffer it is writing to when it completes.
With line buffering (terminal output), the order is:
printf()
prints a line with newline, buffer is flushed, you see1. output from printf()
being printed.echo
writes output, exits, flushes the buffer, you see2. output from a command called using system()
printed.
With block buffering, the order is:
printf()
prints a line with newline, not fully filling the buffer.echo
writes output, exits, flushes the buffer, you see2. output from a command called using system()
printed.- Your program exits, flushes its block buffer, you see
1. output from printf()
being printed.
Your options are to use to flush explicitly using fflush()
or to set the buffering on stdout
explicitly with setvbuf()
.
Why does `` redirect not capture substituted processes' stdout?
The shell that runs head
is spawned by the same shell that runs tee
, which means tee
and head
both inherit the same file descriptor for standard output, which file descriptor is connected to the pipe to cat
. That means both tee
and head
have their output piped to cat
, resulting in the behavior you see.
What does 2&1 mean?
File descriptor 1 is the standard output (stdout
).
File descriptor 2 is the standard error (stderr
).
At first, 2>1
may look like a good way to redirect stderr
to stdout
. However, it will actually be interpreted as "redirect stderr
to a file named 1
".
&
indicates that what follows and precedes is a file descriptor, and not a filename. Thus, we use 2>&1
. Consider >&
to be a redirect merger operator.
Related Topics
Linux: Disable Using Loopback and Send Data via Wire Between 2 Eth Cards of One Comp
The Repository 'Http://Dl.Google.Com/Linux/Chrome/Deb Stable Release' Is Not Signed
Implicit Declaration of Function 'Create_Proc_Entry'
What Is Eof!! in The Bash Script
How to Programmatically Invert Screen Colors in Linux
How to Let Users Run a Script with Root Permissions
Append to /Etc/Apt/Sources.List
Open Eclipse Juno with Error "No More Handles ..." in Ubuntu 12.04
Programmatic Resource Monitoring Per Process in Linux
Redirecting Apache Logs to Stdout
Qemu on Raspberry Pi Arch Linux Latest Sd Image
Differences Between Arm "Versions" (Armv7 Only)
Sudo User Not Using Same Node Version
Gnu Linker: Alternative to -Version-Script to List Exported Symbols at The Command Line
How to Get Maximum Tcp Receive/Send Window in MAC Os X
Transpose Rows into Column in Unix
Docker Run a Shell Script in The Background Without Exiting The Container