Linux/Perl: Additional Output Buffers Other Than Stdout and Stderr

Linux/Perl: Additional output buffers other than STDOUT and STDERR?

Absolutely. The open command with the >&= mode allows you to open filehandles on arbitrary file descriptors.

# perl 4fd.pl > file1 2> file2 3> file3 4> file4 5< file5

open NONSTDFOO, '>&=3';
open NONSTDBAR, '>&=4';
open NONSTDBAZ, '<&=5';   # works for input handles, too

print STDOUT "hello\n";
print STDERR "world\n";
print NONSTDFOO "42\n";
print NONSTDBAR <NONSTDBAZ>;

---

$ echo pppbbbttt > file5
$ perl 4fd.pl >file1 2>file2 3>file3 4>file4 5<file5
$ cat file1
hello
$ cat file3
42
$ cat file4 file2
pppbbbttt
world

PERL Unable to redirect stdout and stderr to output file

Are you Suffering from Buffering?

If so, the data will eventually appear in the log file (unless the program is killed), but you can make it appear there sooner by adding the following to the top of your Perl code:

$| = 1;

The special variable $| (also called $OUTPUT_AUTOFLUSH) will force output to be flushed with every write if its value is non-zero. Follow the link to see how to set autoflush per filehandle.

Why output on perl eval differ between common bash output and redirection STDOUT + STDERR into file?

A simpler demonstration of the problem:

$ perl -e'print("abc\n"); warn("def\n");'
abc
def

$ perl -e'print("abc\n"); warn("def\n");' 2>&1 | cat
def
abc

This is due to differences in how STDOUT and STDERR are buffered.

• STDERR isn't buffered.
• STDOUT flushes its buffer when a newline is encountered if STDOUT is connected to a terminal.
• STDOUT flushes its buffer when it's full otherwise.

$| = 1; turns off buffering for STDOUT^[1].

$ perl -e'$| = 1; print("abc\n"); warn("def\n");' 2>&1 | cat
abc
def

---

1. Actually, the currently selected handle, which is the one print writes to if no handle is specified, which is STDOUT by default.

How do I read and write large buffers to a process stdin/stdout/stderr in Perl?

Since no one managed to make this work I've looked harder in CPAN and found the IPC::Run module.

use IPC::Run;
sub run_prog {
    my ($in, $t, $cmd, @args) = @_;
    my ($out, $err);
    IPC::Run::run([$cmd, @args], \$in, \$out, \$err, IPC::Run::timeout($t));
    return ($?, $out, $err);    
}

Getting STDOUT, STDERR, and response code from external *nix command in perl

Actually, the proper way to write this is:

#!/usr/bin/perl
$cmd = 'lsss';  
my $out=qx($cmd 2>&1);
my $r_c=$?;
print "output was $out\n";
print "return code = ", $r_c, "\n";

You will get a '0' if no error and '-1' if error.

Perl: retrieve output from process in IPC::Run if it dies

As noted by @ysth, the reason you do not get any output, is that the STDOUT and STDERR of the process corresponding to the command $cmd, is not line buffered, but rather block buffered. So all output is collected in a buffer which is not shown (printed) until the buffer is full or it is explicitly flushed. However, when your command times out, all the output is still in the buffer and has not yet been flushed and hence collected into the variable $out in the parent process (script).

Also note that since your $cmd script is a Perl script, this behavior is documented in perlvar:

$|

If set to nonzero, forces a flush right away and after every write
or print on the currently selected output channel. Default is 0
(regardless of whether the channel is really buffered by the system or
not; $| tells you only whether you've asked Perl explicitly to flush
after each write). STDOUT will typically be line buffered if output is
to the terminal and block buffered otherwise.

The problem (that the program is not connected to a terminal or a tty) is also noted in the documentation page for IPC::Run :

Interactive applications are usually optimized for human use. This can
help or hinder trying to interact with them through modules like
IPC::Run. Frequently, programs alter their behavior when they detect
that stdin, stdout, or stderr are not connected to a tty, assuming
that they are being run in batch mode. Whether this helps or hurts
depends on which optimizations change. And there's often no way of
telling what a program does in these areas other than trial and error
and occasionally, reading the source. This includes different versions
and implementations of the same program.

The documentation also lists a set of possible workarounds, including using pseudo terminals.

One solution for your specific case is then to explicitly make STDOUT line buffered at the beginning of your script:

STDOUT->autoflush(1);  # Make STDOUT line buffered
# Alternatively use: $| = 1; 
for (my $i = 0; $i < 10; $i++) {
  sleep 1;
  print "Hello from script 1 " . localtime() . "\n";
}

Edit:

If you cannot modify the scripts you are running for some reason, you could try connect the script to a pseudo terminal. So instead of inserting statements like STDOUT->autoflush(1) in the source code of the script, you can fool the script to believe it is connected to a terminal, and hence that it should use line buffering. For your case, we just add a >pty> argument before the \$out argument in the call to harness:

my $h = harness $cmd, \undef, '>pty>', \$out,
  timeout(12, exception => {name => 'timeout'});
eval {
    run $h;
};

---

Related Topics