How to 'grep' a continuous stream?
Turn on grep
's line buffering mode when using BSD grep (FreeBSD, Mac OS X etc.)
tail -f file | grep --line-buffered my_pattern
It looks like a while ago --line-buffered
didn't matter for GNU grep (used on pretty much any Linux) as it flushed by default (YMMV for other Unix-likes such as SmartOS, AIX or QNX). However, as of November 2020, --line-buffered
is needed (at least with GNU grep 3.5 in openSUSE, but it seems generally needed based on comments below).
How to process continuous stream output with grep utility?
As suggested by @larsks , " --line-buffered flush output on every line" option for grep is working fine when is test for similar requirement as yous .
So the command would be
curl -X "POST" "http://localhost:8088/query" --header "Content-Type: application/json" -d $'{"ksql": "select * from SENSOR_S EMIT CHANGES;","streamsProperties": {"ksql.streams.auto.offset.reset": "earliest"}}' -s -N | grep S1 --line-buffered | xargs -I {} echo {}
I tested on "/var/log/messages" file which gets continously updated as following :
[root@project1-master ~]# tail -f /var/log/messages | grep journal --line-buffered | xargs -I {} echo {}
Sep 11 11:15:47 project1-master journal: I0911 15:15:47.448254 1 node_lifecycle_controller.go:1429] Initializing eviction metric for zone:
Sep 11 11:15:52 project1-master journal: I0911 15:15:52.448704 1 node_lifecycle_controller.go:1429] Initializing eviction metric for zone:
Sep 11 11:15:54 project1-master journal: 2020-09-11 15:15:54.006 [INFO][46] felix/int_dataplane.go 1300: Applying dataplane updates
How do I grep or sed in a continuous stream of characters?
The issue here is that both grep
and sed
process their input linewise unless they are told otherwise. One workaround is to insert newlines, e.g. with coreutils fold
:
grep -m 1 A \
<(perl -E '$|=1;print "OUT\n";print "A" and sleep 1 for 1..10' | fold -w1)
But this still has issues because of pipe-buffering. You can disable this with coreutils stdbuf
:
grep -m 1 A \
<(perl -E '$|=1;print "OUT\n";print "A" and sleep 1 for 1..10' |
stdbuf -o0 fold -w1)
Another alternative is to use dd
, e.g.:
perl -E '$|=1;print "OUT\n";print "A" and sleep 1 for 1..10' |
while :; do
c=$(dd bs=1 count=1 status=none)
if [[ "$c" == "A" ]]; then
break
else
: Do something else
fi
done
Linux bash: grep from stream and write to file
If just grep
, without writing to the file, works, you encountered a buffering "problem". I/O buffering, unless manually implemented by the program will get handled by the libc. If the program's stdout is a termial, buffering will be line-based. If not, the libc buffers output until the buffer reached a size limit.
On Linux, meaning with glibc
you can use the stdbuf
command to configure that buffering:
tail -f A.log | stdbuf -oL grep "keyword" >> B.log
-oL
specifies that the output stream should be line-buffered.
Related Topics
Multiple Glibc Libraries on a Single Host
How to Prevent a Background Process from Being Stopped After Closing Ssh Client in Linux
How to Get Process Id of Background Process
Static Link of Shared Library Function in Gcc
Find and Replace With Sed in Directory and Sub Directories
How to Get Terminal'S Character Encoding
Get Most Recent File in a Directory on Linux
Is Gettimeofday() Guaranteed to Be of Microsecond Resolution
How Does Mmap Improve File Reading Speed
How to Get Full Path of a File
Looping Through the Content of a File in Bash
How Do the Likely/Unlikely Macros in the Linux Kernel Work and What Is Their Benefit
How to Get Overall Cpu Usage (E.G. 57%) on Linux
"Unable to Find Remote Helper For 'Https'" During Git Clone
Appending a Line to a File Only If It Does Not Already Exist
How to View the List of Functions a Linux Shared Library Is Exporting