How do I write standard error to a file while using tee with a pipe?
I'm assuming you want to still see standard error and standard output on the terminal. You could go for Josh Kelley's answer, but I find keeping a tail
around in the background which outputs your log file very hackish and cludgy. Notice how you need to keep an extra file descriptor and do cleanup afterward by killing it and technically should be doing that in a trap '...' EXIT
.
There is a better way to do this, and you've already discovered it: tee
.
Only, instead of just using it for your standard output, have a tee for standard output and one for standard error. How will you accomplish this? Process substitution and file redirection:
command > >(tee -a stdout.log) 2> >(tee -a stderr.log >&2)
Let's split it up and explain:
> >(..)
>(...)
(process substitution) creates a FIFO and lets tee
listen on it. Then, it uses >
(file redirection) to redirect the standard output of command
to the FIFO that your first tee
is listening on.
The same thing for the second:
2> >(tee -a stderr.log >&2)
We use process substitution again to make a tee
process that reads from standard input and dumps it into stderr.log
. tee
outputs its input back on standard output, but since its input is our standard error, we want to redirect tee
's standard output to our standard error again. Then we use file redirection to redirect command
's standard error to the FIFO's input (tee
's standard input).
See Input And Output
Process substitution is one of those really lovely things you get as a bonus of choosing Bash as your shell as opposed to sh
(POSIX or Bourne).
In sh
, you'd have to do things manually:
out="${TMPDIR:-/tmp}/out.$$" err="${TMPDIR:-/tmp}/err.$$"
mkfifo "$out" "$err"
trap 'rm "$out" "$err"' EXIT
tee -a stdout.log < "$out" &
tee -a stderr.log < "$err" >&2 &
command >"$out" 2>"$err"
Why would tee fail to write to stdout?
You aren't misunderstanding tee
, you're misunderstanding what stdout is. In a pipe, like echo testing | tee | python3 -c 'a=1'
, the tee
command's stdout is not the terminal, it's the pipe going to the python
command (and the echo
command's stdout is the pipe going to tee
).
So tee /dev/stdout
sends two copies of its input (on stdin) to the exact same place: its stdout, whether that's the terminal, or a pipe, or whatever.
If you want to send a copy of the input to tee
someplace other than down the pipe, you need to send it somewhere other than stdout. Where that is depends on where you actually want to send it (i.e. why you want to copy it). If you specifically want to send it to the terminal, you could do this:
echo testing | tee /dev/tty | python3 -c 'a=1'
...while if you want to send it to the outer context's stdout (which might or might not be a terminal), you can duplicate the outer context's stdin to a different file descriptor (#3 is handy for this), and then have tee
write a copy to that:
{ echo testing | tee /dev/fd/3 | python3 -c 'a=1'; } 3>&1
Yet another option is to redirect it to stderr (aka FD #2, which is also the terminal by default, but redirectable separately from stdout) with tee /dev/fd/2
.
Note that the various /dev entries I'm using here are supported by most unixish OSes, but they aren't universal. Check to see what your specific OS provides.
Linux: tee usage misunderstanding
TL;DR -- Try this one:
svnadmin dump MyRepo -r1:r2 2>&1 > dumpfile | tee log.txt
Explanation: Each command has three connected "standard streams": Standard Input (STDIN, filehandle 0), Standard Output (STDOUT, 1) and Standard Error (STDERR, 2). Usually commands ouput their "data" on STDOUT and errors on STDERR.
A simple command like ls
has all three of them connected to the console. Because both STDOUT and STDERR are connected to the console the output of the command is interleaved.
A "pipe" like ls | tee log.txt
redirects STDOUT of the first command to STDIN of the second command. No more - no less. Therefore all other streams are still connected to the console. Should the ls
part produce error messages they will be written to the console, not to the file! But your ls
example did not output any errors so you didn't notice.
After the pipe is setup the shell evaluates the other redirection operator of the first command -- from left to right.
Therefore svnadmin dump > dumpfile | tee log.txt
will redirect STDOUT of svnadmin
to dumpfile
leaving effectively no data for the tee
command because that's a redirection, not a copy.
svnadmin dump MyRepo 2>&1 > dumpfile | tee log.txt
adds another redirection step. It reads "make filehandle 2 (STDERR) a copy of the filehandle 1 (STDOUT)" which is the pipe at this point. Writing to either STDOUT or STDERR will write to the pipe. But after that the > dumpfile
redirection is applied and STDOUT is redirected to the file.
You can read all this (and much more) in the shell's manual. For bash
it is in the section REDIRECTION
.
| vs for stdin and stdout with splice, pipe and tee command in C
The two methods are not equivalent:
cat in.txt |
will actually create apipe
to feed your program< in.txt
will justopen
the given file and serve it afd
0
On second case, tee
returns -1
, and perror
complains about an invalid argument.
You can check the nature of fd
0
using fstat
and checking the st_mode
value:
struct stat statbuf;
fstat(STDIN_FILENO, &statbuf);
if (statbuf.st_mode & S_IFREG)
printf("regular file");
if (statbuf.st_mode & S_IFIFO)
printf("pipe");
Tee doesn't get the output from the pipe
If you redirect the output to exec > path/logfile.log
then... well, the output will be redirected to the file, not to the pipe.
Try:
#!/bin/bash
{
dirname=/path
tempfile=myTempfileName
find "$dirname" -type f > "$tempfile"
sed 's_.*/__' "$tempfile" | sort | uniq -d |
while read fileName
do
grep "$fileName" "$tempfile"
done
} 2>&1 | tee -a path/logfile.log | tee 'path/scripts/tj_var1.txt'
# ^^ I guess log file should be appended.
I guess you could have only stdout in the tj_var1.txt
file:
#!/bin/bash
{
{
dirname=/path
tempfile=myTempfileName
find "$dirname" -type f > "$tempfile"
sed 's_.*/__' "$tempfile" | sort | uniq -d |
while read fileName
do
grep "$fileName" "$tempfile"
done
} | tee 'path/scripts/tj_var1.txt'
} 2>&1 | tee -a path/logfile.log
It basically finds out the duplicate filenames and outputs it.
Just:
dirname=/path
find "$dirname" -type f -printf "%f\n" |
sort | uniq -d |
tee -a path/logfile.log | tee 'path/scripts/tj_var1.txt'
How to tee to stderr?
./script.sh | tee /dev/fd/2
Note that this is dependant on OS support, not any built-in power in tee, so isn't universal (but will work on MacOS, Linux, Solaris, FreeBSD, probably others).
On what occasion does 'tee' deletes the file it was writing on?
On what occasion does 'tee' deletes the file it was writing on?
tee
does not delete nor truncate the file once it has started writing.
Is there a filesize limit or timeout in tee?
No.
Is there any known behavior of tee that it deletes a file?
No.
Note that file can be removed, but the process (tee
) still will wrote the open file descriptor, but the file will not be accessible (see man 3 unlink
).
Related Topics
Rename All Files in a Folder With a Prefix in a Single Command
How to Detect Invalid Utf8 Unicode/Binary in a Text File
How to Scroll Up/Down on the Console of a Linux Vm
How to Remove Cached Credentials from Git
What Killed My Process and Why
Automatically Enter Ssh Password With Script
How to Send a HTML Email With the Bash Command "Sendmail"
Print a File, Skipping the First X Lines, in Bash
Using Awk to Count the Number of Occurrences of a Word in a Column
How to Reload Google Chrome Tab from Terminal
How to Simulate Just One Enter in Command Line After Executing a Jar File
Curl: (6) Could Not Resolve Host: Google.Com; Name or Service Not Known
How to Get the Bssid of Currently Connected Network Through Bash
Error:13 - Permission Denied Android Studio
Sort a List With Lowercase First