How to 'Grep' a Continuous Stream

How to 'grep' a continuous stream?

Turn on grep's line buffering mode when using BSD grep (FreeBSD, Mac OS X etc.)

tail -f file | grep --line-buffered my_pattern

It looks like a while ago --line-buffered didn't matter for GNU grep (used on pretty much any Linux) as it flushed by default (YMMV for other Unix-likes such as SmartOS, AIX or QNX). However, as of November 2020, --line-buffered is needed (at least with GNU grep 3.5 in openSUSE, but it seems generally needed based on comments below).

How to process continuous stream output with grep utility?

As suggested by @larsks , " --line-buffered flush output on every line" option for grep is working fine when is test for similar requirement as yous .

So the command would be

curl -X "POST" "http://localhost:8088/query" --header "Content-Type: application/json" -d $'{"ksql": "select * from SENSOR_S EMIT CHANGES;","streamsProperties": {"": "earliest"}}' -s -N | grep S1 --line-buffered | xargs -I {} echo {}

I tested on "/var/log/messages" file which gets continously updated as following :

[root@project1-master ~]# tail -f /var/log/messages | grep journal --line-buffered | xargs -I {} echo {}

Sep 11 11:15:47 project1-master journal: I0911 15:15:47.448254 1 node_lifecycle_controller.go:1429] Initializing eviction metric for zone:

Sep 11 11:15:52 project1-master journal: I0911 15:15:52.448704 1 node_lifecycle_controller.go:1429] Initializing eviction metric for zone:

Sep 11 11:15:54 project1-master journal: 2020-09-11 15:15:54.006 [INFO][46] felix/int_dataplane.go 1300: Applying dataplane updates

How do I grep or sed in a continuous stream of characters?

The issue here is that both grep and sed process their input linewise unless they are told otherwise. One workaround is to insert newlines, e.g. with coreutils fold:

grep -m 1 A \
<(perl -E '$|=1;print "OUT\n";print "A" and sleep 1 for 1..10' | fold -w1)

But this still has issues because of pipe-buffering. You can disable this with coreutils stdbuf:

grep -m 1 A \
<(perl -E '$|=1;print "OUT\n";print "A" and sleep 1 for 1..10' |
stdbuf -o0 fold -w1)

Another alternative is to use dd, e.g.:

perl -E '$|=1;print "OUT\n";print "A" and sleep 1 for 1..10' |
while :; do
c=$(dd bs=1 count=1 status=none)
if [[ "$c" == "A" ]]; then
: Do something else

Linux bash: grep from stream and write to file

If just grep, without writing to the file, works, you encountered a buffering "problem". I/O buffering, unless manually implemented by the program will get handled by the libc. If the program's stdout is a termial, buffering will be line-based. If not, the libc buffers output until the buffer reached a size limit.

On Linux, meaning with glibc you can use the stdbuf command to configure that buffering:

tail -f A.log | stdbuf -oL grep "keyword" >> B.log

-oL specifies that the output stream should be line-buffered.

Related Topics

Leave a reply