Grep with Continuous Pipe Does Not Work

grep with continuous pipe does not work

I think the problem is because of stdio buffering, you need to use GNU stdbuf before calling grep,

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | stdbuf -o0 grep '^Host: '

With the -o0, it basically means the output (stdout) stream from tcpflow will be unbuffered. The default behavior will be to automatically buffer up data into 4096¹ byte chunks before sending to next command in pipeline, which is what overriden using stdbuf

---

_{1. Refer this nice detail into the subject.}

Piping of grep is not working with tail?

grep is buffering it's output. To tell GNU grep to spit out output line-by-line you need to use --line-buffered option in grep to make it work:

tail -f eclipse.log | grep --line-buffered 'enimation' | grep --line-buffered -i 'tap'

As per man grep:

--line-buffered
    Force output to be line buffered.  By default, output is line buffered when standard
    output is a terminal and block buffered otherwise.

How to 'grep' a continuous stream?

Turn on grep's line buffering mode when using BSD grep (FreeBSD, Mac OS X etc.)

tail -f file | grep --line-buffered my_pattern

It looks like a while ago --line-buffered didn't matter for GNU grep (used on pretty much any Linux) as it flushed by default (YMMV for other Unix-likes such as SmartOS, AIX or QNX). However, as of November 2020, --line-buffered is needed (at least with GNU grep 3.5 in openSUSE, but it seems generally needed based on comments below).

Unable to pipe output after grep

For grepping stream (tail -f)

it's almost always better to use grep --line-buffered instead of just grep

From official docs:

--line-buffered Use line buffering on output. This can cause a performance penalty.

In other words: Write output each time when grep get input and do not wait for filling grep buffer

bash: pipe continuously into a grep

What's Going Wrong

The code: if grep patternB; then echo "true"; fi

...waits for grep patternB to exit, which will happen only when the input from tail -f file.log | grep patternA hits EOF. Since tail -f waits for new content forever, there will never be an EOF, so your if statement will never complete.

How To Fix It

Don't use grep on the inside of your function. Instead, process content line-by-line and use bash's native regex support:

customGrepFunction() {
  while IFS= read -r line; do
    if [[ $line =~ patternB ]]; then
      echo "True"
    fi
  done
}

Next, make sure that grep isn't buffering content (if it were, then it would be written to your code only in big chunks, delaying until such a chunk is available). The means to do this varies by implementation, but with GNU grep, it would look like:

tail -f file.log | grep --line-buffered patternA | customGrepFunction

No output while piping tail through grep

grep is not line-buffered when its stdout isn't connected to a tty. So it's trying to process a block (usually 4 KiB or 8 KiB or so) before generating some output.

You need to tell grep to buffer its output by line. If you're using GNU grep, this works:

done < <(tail -F -n +1 "$filepath" | grep '^!' --line-buffered)
                                               ^^^^^^^^^^^^^^^

How to process continuous stream output with grep utility?

As suggested by @larsks , " --line-buffered flush output on every line" option for grep is working fine when is test for similar requirement as yous .

So the command would be

curl -X "POST" "http://localhost:8088/query" --header "Content-Type: application/json" -d $'{"ksql": "select * from SENSOR_S EMIT CHANGES;","streamsProperties": {"ksql.streams.auto.offset.reset": "earliest"}}' -s -N | grep S1 --line-buffered | xargs -I {} echo {}

I tested on "/var/log/messages" file which gets continously updated as following :

[root@project1-master ~]# tail -f /var/log/messages | grep journal --line-buffered | xargs -I {} echo {}

Sep 11 11:15:47 project1-master journal: I0911 15:15:47.448254 1 node_lifecycle_controller.go:1429] Initializing eviction metric for zone:

Sep 11 11:15:52 project1-master journal: I0911 15:15:52.448704 1 node_lifecycle_controller.go:1429] Initializing eviction metric for zone:

Sep 11 11:15:54 project1-master journal: 2020-09-11 15:15:54.006 [INFO][46] felix/int_dataplane.go 1300: Applying dataplane updates

Why no output is shown when using grep twice?

You might also run into a problem with grep buffering when inside a pipe.
ie, you don't see the output from

   tail --follow=name file.txt | grep something > output.txt

since grep will buffer its own output.

Use the --line-buffered switch for grep to work around this:

tail --follow=name file.txt | grep --line-buffered something > output.txt

This is useful if you want to get the results of the follow into the output.txt file as rapidly as possible.

---

Related Topics