How to Disable Stdout Buffer When Running Shell

Turn off buffering in pipe

You can use the unbuffer command (which comes as part of the expect package), e.g.

unbuffer long_running_command | print_progress

unbuffer connects to long_running_command via a pseudoterminal (pty), which makes the system treat it as an interactive process, therefore not using the 4-kiB buffering in the pipeline that is the likely cause of the delay.

For longer pipelines, you may have to unbuffer each command (except the final one), e.g.

unbuffer x | unbuffer -p y | z

Disable output buffering

From Magnus Lycka answer on a mailing list:

You can skip buffering for a whole
python process using python -u
(or #!/usr/bin/env python -u etc.) or by
setting the environment variable
PYTHONUNBUFFERED.

You could also replace sys.stdout with
some other stream like wrapper which
does a flush after every call.

class Unbuffered(object):
   def __init__(self, stream):
       self.stream = stream
   def write(self, data):
       self.stream.write(data)
       self.stream.flush()
   def writelines(self, datas):
       self.stream.writelines(datas)
       self.stream.flush()
   def __getattr__(self, attr):
       return getattr(self.stream, attr)

import sys
sys.stdout = Unbuffered(sys.stdout)
print 'Hello'

How to make output of any shell command unbuffered?

AFAIK, you can't do it without ugly hacks. Writing to a pipe (or reading from it) automatically turns on full buffering and there is nothing you can do about it :-(. "Line buffering" (which is what you want) is only used when reading/writing a terminal. The ugly hacks exactly do this: They connect a program to a pseudo-terminal, so that the other tools in the pipe read/write from that terminal in line buffering mode. The whole problem is described here:

• http://www.pixelbeat.org/programming/stdio_buffering/

The page has also some suggestions (the aforementioned "ugly hacks") what to do, i.e. using unbuffer or pulling some tricks with LD_PRELOAD.

How to disable output buffering in Process.StandardOutput

"[I]s there a way to launch it such [that] it doesn't realize it is being redirected?" YES: that is exactly the domain of Expect. I know of no .Net implementation; it's surely feasible, though ...

Disabling stdout buffering of a forked process

Actually, after struggling with it a bit, it seems like the only solution to this problem is by making the 'parent' process pretending to be a terminal using the OS pseudo terminal API calls.

One should call 'openpty()' before the fork(), and inside the child code, he should call 'login_tty(slave)' the slave is then becoming the stdin/out and stderr.

By pretending to a terminal, the buffering of stdout is automatically set to 'line mode' (i.e. flush occurs when \n is encountered). The parent should use the 'master' descriptor for readin/writing with the child process.

The modified parent code (in case anyone will ever need this):

#include <unistd.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/select.h>
#include <errno.h>
#include <sys/wait.h>
#include <string>
#include <string.h>
#include <cstdio>
#include <pty.h>
#include <utmp.h>
static int   read_handle(-1);
static pid_t pid;

bool read_from_child(std::string& buff) {
    fd_set  rs;
    timeval timeout;

    memset(&rs, 0, sizeof(rs));
    FD_SET(read_handle, &rs);
    timeout.tv_sec  = 1; // 1 second
    timeout.tv_usec = 0;

    int rc = select(read_handle+1, &rs, NULL, NULL, &timeout);
    if ( rc == 0 ) {
        // timeout
        return true;

    } else if ( rc > 0 ) {
        // there is something to read
        char buffer[1024*64]; // our read buffer
        memset(buffer, 0, sizeof(buffer));
        if(read(read_handle, buffer, sizeof(buffer)) > 0) {
            buff.clear();
            buff.append( buffer );
            return true;
        }

        return false;
    } else { /* == 0 */
        if ( rc == EINTR || rc == EAGAIN ) {
            return true;
        }

        // Process terminated
        int status(0);
        waitpid(pid, &status, 0);
        return false;
    }
}

void execute() {
    char *argv[] = {"/home/eran/devl/TestMain/Debug/TestMain", NULL};
    int    argc = 1;

    int master, slave;
    openpty(&master, &slave, NULL, NULL, NULL);

    int rc = fork();
    if ( rc == 0 ) {
        login_tty(slave);
        close(master);

        // execute the process
        if(execvp(argv[0], argv) != 0)
            perror("execvp");

    } else if ( rc < 0 ) {
        perror("fork");
        return;

    } else {
        // Parent
        std::string buf;
        close(slave);

        read_handle = master;
        while(read_from_child(buf)) {
            if(buf.empty() == false) {
                printf("Received: %s", buf.c_str());
            }
            buf.clear();
        }
    }
}

int main(int argc, char **argv) {
    execute();
    return 0;
}

Force line-buffering of stdout in a pipeline

Try unbuffer (man page) which is part of the expect package. You may already have it on your system.

In your case you would use it like this:

unbuffer ./a | tee output.txt

The -p option is for pipeline mode where unbuffer reads from stdin and passes it to the command in the rest of the arguments.

---

Related Topics