How Can a Process Intercept Stdout and Stderr of Another Process on Linux

How can a process intercept stdout and stderr of another process on Linux?

I'm not sure if it will work for you, but I read a page a while back describing a method that uses gdb

Redirect STDERR / STDOUT of a process AFTER it's been started, using command line?

Short of closing and reopening your tty (i.e. logging off and back on, which may also terminate some of your background processes in the process) you only have one choice left:

  • attach to the process in question using gdb, and run:

    • p dup2(open("/dev/null", 0), 1)
    • p dup2(open("/dev/null", 0), 2)
    • detach
    • quit

e.g.:

$ tail -f /var/log/lastlog &
[1] 5636

$ ls -l /proc/5636/fd
total 0
lrwx------ 1 myuser myuser 64 Feb 27 07:36 0 -> /dev/pts/0
lrwx------ 1 myuser myuser 64 Feb 27 07:36 1 -> /dev/pts/0
lrwx------ 1 myuser myuser 64 Feb 27 07:36 2 -> /dev/pts/0
lr-x------ 1 myuser myuser 64 Feb 27 07:36 3 -> /var/log/lastlog

$ gdb -p 5636
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Attaching to process 5636
Reading symbols from /usr/bin/tail...(no debugging symbols found)...done.
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x7f3c8f5a66e0 (LWP 5636)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2

(no debugging symbols found)
0x00007f3c8eec7b50 in nanosleep () from /lib/libc.so.6

(gdb) p dup2(open("/dev/null",0),1)
[Switching to Thread 0x7f3c8f5a66e0 (LWP 5636)]
$1 = 1

(gdb) p dup2(open("/dev/null",0),2)
$2 = 2

(gdb) detach
Detaching from program: /usr/bin/tail, process 5636

(gdb) quit

$ ls -l /proc/5636/fd
total 0
lrwx------ 1 myuser myuser 64 Feb 27 07:36 0 -> /dev/pts/0
lrwx------ 1 myuser myuser 64 Feb 27 07:36 1 -> /dev/null
lrwx------ 1 myuser myuser 64 Feb 27 07:36 2 -> /dev/null
lr-x------ 1 myuser myuser 64 Feb 27 07:36 3 -> /var/log/lastlog
lr-x------ 1 myuser myuser 64 Feb 27 07:36 4 -> /dev/null
lr-x------ 1 myuser myuser 64 Feb 27 07:36 5 -> /dev/null

You may also consider:

  • using screen; screen provides several virtual TTYs you can switch between without having to open new SSH/telnet/etc, sessions
  • using nohup; this allows you to close and reopen your session without losing any background processes in the... process.

observing stdout of another process

http://ingvar.blog.linpro.no/2010/07/10/changing-a-process-file-descriptor-on-the-fly/

How to redirect output of an already running process

See Redirecting Output from a Running Process.

Firstly I run the command cat > foo1 in one session and test that data from stdin is copied to the file. Then in another session I redirect the output.

Firstly find the PID of the process:

$ ps aux | grep cat
rjc 6760 0.0 0.0 1580 376 pts/5 S+ 15:31 0:00 cat

Now check the file handles it has open:

$ ls -l /proc/6760/fd
total 3
lrwx—— 1 rjc rjc 64 Feb 27 15:32 0 -> /dev/pts/5
l-wx—— 1 rjc rjc 64 Feb 27 15:32 1 -> /tmp/foo1
lrwx—— 1 rjc rjc 64 Feb 27 15:32 2 -> /dev/pts/5

Now run GDB:

$ gdb -p 6760 /bin/cat
GNU gdb 6.4.90-debian

[license stuff snipped]

Attaching to program: /bin/cat, process 6760

[snip other stuff that's not interesting now]

(gdb) p close(1)
$1 = 0
(gdb) p creat("/tmp/foo3", 0600)
$2 = 1
(gdb) q
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: /bin/cat, process 6760

The p command in GDB will print the value of an expression, an expression can be a function to call, it can be a system call… So I execute a close() system call and pass file handle 1, then I execute a creat() system call to open a new file. The result of the creat() was 1 which means that it replaced the previous file handle. If I wanted to use the same file for stdout and stderr or if I wanted to replace a file handle with some other number then I would need to call the dup2() system call to achieve that result.

For this example I chose to use creat() instead of open() because there are fewer parameter. The C macros for the flags are not usable from GDB (it doesn’t use C headers) so I would have to read header files to discover this – it’s not that hard to do so but would take more time. Note that 0600 is the octal permission for the owner having read/write access and the group and others having no access. It would also work to use 0 for that parameter and run chmod on the file later on.

After that I verify the result:

ls -l /proc/6760/fd/
total 3
lrwx—— 1 rjc rjc 64 2008-02-27 15:32 0 -> /dev/pts/5
l-wx—— 1 rjc rjc 64 2008-02-27 15:32 1 -> /tmp/foo3 <====
lrwx—— 1 rjc rjc 64 2008-02-27 15:32 2 -> /dev/pts/5

Typing more data in to cat results in the file /tmp/foo3 being appended to.

If you want to close the original session you need to close all file handles for it, open a new device that can be the controlling tty, and then call setsid().

Separately redirecting and recombining stderr/stdout without losing ordering

Preserving perfect order while performing separate redirections is not even theoretically possible without some ugly hackery. Ordering is only preserved in writes (in O_APPEND mode) directly to the same file; as soon as you put something like tee in one process but not the other, ordering guarantees go out the window and can't be retrieved without keeping information about which syscalls were invoked in what order.

So, what would that hackery look like? It might look something like this:

# eat our initialization time *before* we start the background process
sudo sysdig-probe-loader

# now, start monitoring syscalls made by children of this shell that write to fd 1 or 2
# ...funnel content into our logs.log file
sudo sysdig -s 32768 -b -p '%evt.buffer' \
"proc.apid=$$ and evt.type=write and (fd.num=1 or fd.num=2)" \
> >(base64 -i -d >logs.log) \
& sysdig_pid=$!

# Run your-program, with stderr going both to console and to errors.log
./your-program >/dev/null 2> >(tee errors.log)

That said, this remains ugly hackery: It only catches writes direct to FDs 1 and 2, and doesn't track any further redirections that may take place. (This could be improved by performing the writes to FIFOs, and using sysdig to track writes to those FIFOs; that way fdup() and similar operations would work as-expected; but the above suffices to prove the concept).


Making Separate Handling Explicit

Here we demonstrate how to use this to colorize only stderr, and leave stdout alone -- by telling sysdig to generate a stream of JSON as output, and then iterating over that:

exec {colorizer_fd}> >(
jq --unbuffered --arg startColor "$(tput setaf 1)" --arg endColor "$(tput sgr0)" -r '
if .["fd.filename"] == "stdout" then
("STDOUT: " + .["evt.buffer"])
else
("STDERR: " + $startColor + .["evt.buffer"] + $endColor)
end
'
)

sudo sysdig -s 32768 -j -p '%fd.filename %evt.buffer' \
"proc.apid=$$ and evt.type=write and proc.name != jq and (fd.num=1 or fd.num=2)" \
>&$colorizer_fd \
& sysdig_pid=$!

# Run your-program, with stdout and stderr going to two separately-named destinations
./your-program >stdout 2>stderr

Because we're keying off the output filenames (stdout and stderr), these need to be constant for the above code to work -- any temporary directory desired can be used.


Obviously, you shouldn't actually do any of this. Update your program to support whatever logging infrastructure is available in its native language (Log4j in Java, the Python logging module, etc) to allow its logging to be configured explicitly.



Related Topics



Leave a reply



Submit