Why No Output Is Shown When Using Grep Twice

Why no output is shown when using grep twice?

You might also run into a problem with grep buffering when inside a pipe.
ie, you don't see the output from

   tail --follow=name file.txt | grep something > output.txt

since grep will buffer its own output.

Use the --line-buffered switch for grep to work around this:

tail --follow=name file.txt | grep --line-buffered something > output.txt

This is useful if you want to get the results of the follow into the output.txt file as rapidly as possible.

Piping tail output though grep twice

I believe the problem here is that the first grep is buffering the output which means the second grep won't see it until the buffer is flushed.

Try adding the --line-buffered option on your first grep:

tail -f access_log | grep --line-buffered "127.0.0.1" | grep -v ".css"

For more info, see "BashFAQ/009 -- What is buffering? Or, why does my command line produce no output: tail -f logfile | grep 'foo bar' | awk ..."

No output while piping tail through grep

grep is not line-buffered when its stdout isn't connected to a tty. So it's trying to process a block (usually 4 KiB or 8 KiB or so) before generating some output.

You need to tell grep to buffer its output by line. If you're using GNU grep, this works:

done < <(tail -F -n +1 "$filepath" | grep '^!' --line-buffered)
                                               ^^^^^^^^^^^^^^^

Is it more efficient to grep twice or use a regular expression once?

grep -E '(foo|bar)' will find lines containing 'foo' OR 'bar'.

You want lines containing BOTH 'foo' AND 'bar'. Either of these commands will do:

sed '/foo/!d;/bar/!d' file.log

awk '/foo/ && /bar/' file.log

Both commands -- in theory -- should be much more efficient than your cat | grep | grep construct because:

• Both sed and awk perform their own file reading; no need for pipe overhead
• The 'programs' I gave to sed and awk above use Boolean short-circuiting to quickly skip lines not containing 'foo', thus testing only lines containing 'foo' to the /bar/ regex

However, I haven't tested them. YMMV :)

Why can't I filter tail's output multiple times through pipes?

It's a buffering issue - the first grep buffers it's output when it's piping to another command but not if it's printing to stdout. See http://mywiki.wooledge.org/BashFAQ/009 for additional info.

Why does grep output a different result based on the terminal width?

My suspicion is that your output contains Carriage Return (CR) characters (0x0D, \r) which moves the cursor to the beginning of the line without advancing to the next line. This is the beginning of the line in the terminal, not the beginning of the output line. From your output, it looks like there is one in front of the word [blackdetect @ 0x7fb80909300] but it is likely that there are more.

You can visualize the CR using cat -vET where it will be converted to a ^M combination.

$ command | cat -vET`

You can see the effects of a CR and the terminal width with the following simple example:

$ a=$(printf "%s" {0..9})
$ echo $a$a$a$'\r'$a

With a terminal width of 23, the output looks like:

012345678901234567890123
0123456789

where you clearly see that the characters 456789 are missing from the second line.

Related: How to replace carriage return without new line in FFmpeg output thanks to sed?

---

Related Topics