Awk, Pipe and Tail -F Giving Unexpected Behavior

Awk, pipe and tail -f giving unexpected behavior

When the output of the first awk is going to the terminal, the output is line-buffered, so each line is printed as it is produced. When the output is going to the second awk or the grep, it is fully buffered. The output won't be sent until the buffer is full. When enough extra records are appended to the log, then the second awk will a buffer full of data to process. Until then, nothing will happen.

Piping tail output though grep twice

I believe the problem here is that the first grep is buffering the output which means the second grep won't see it until the buffer is flushed.

Try adding the --line-buffered option on your first grep:

tail -f access_log | grep --line-buffered "127.0.0.1" | grep -v ".css"

For more info, see "BashFAQ/009 -- What is buffering? Or, why does my command line produce no output: tail -f logfile | grep 'foo bar' | awk ..."

Do a tail -F until matching a pattern

Try this:

sh -c 'tail -n +0 -f /tmp/foo | { sed "/EOF/ q" && kill $$ ;}'

The whole command-line will exit as soon as the "EOF" string is seen in /tmp/foo.

There is one side-effect: the tail process will be left running (in the background) until anything is written to /tmp/foo.

Bash: Head & Tail behavior with bash script

This is a fairly interesting issue! Thanks for posting it!

I assumed that this happens as head exits after processing the first few lines, so SIGPIPE signal is sent to the bash running the script when it tries to echo $x next time. I used RedX's script to prove this theory:

#!/usr/bin/bash
rm x.log
for((x=0;x<5;++x)); do
    echo $x
    echo $x>>x.log
done

This works, as You described! Using t.sh|head -n 2 it writes only 2 lines to the screen and to x.log. But trapping SIGPIPE this behavior changes...

#!/usr/bin/bash
trap "echo SIGPIPE>&2" PIPE
rm x.log
for((x=0;x<5;++x)); do
    echo $x
    echo $x>>x.log
done

Output:

$ ./t.sh |head -n 2
0
1
./t.sh: line 5: echo: write error: Broken pipe
SIGPIPE
./t.sh: line 5: echo: write error: Broken pipe
SIGPIPE
./t.sh: line 5: echo: write error: Broken pipe
SIGPIPE

The write error occurs as stdout is already closed as the other end of the pipe is closed. And any attempt to write to the closed pipe causes a SIGPIPE signal, which terminates the program by default (see man 7 signal). The x.log now contains 5 lines.

This also explains why /bin/echo solved the problem. See the following script:

rm x.log
for((x=0;x<5;++x)); do
    /bin/echo $x
    echo "Ret: $?">&2
    echo $x>>x.log
done

Output:

$ ./t.sh |head -n 2
0
Ret: 0
1
Ret: 0
Ret: 141
Ret: 141
Ret: 141

Decimal 141 = hex 8D. Hex 80 means a signal was received, hex 0D is for SIGPIPE. So when /bin/echo tried to write to stdout it got a SIGPIPE and it was terminated (as default behavior) instead of the bash running the script.

A variable modified inside a while loop is not remembered

echo -e $lines | while read line 
    ...
done

The while loop is executed in a subshell. So any changes you do to the variable will not be available once the subshell exits.

Instead you can use a here string to re-write the while loop to be in the main shell process; only echo -e $lines will run in a subshell:

while read line
do
    if [[ "$line" == "second line" ]]
    then
        foo=2
        echo "Variable \$foo updated to $foo inside if inside while loop"
    fi
    echo "Value of \$foo in while loop body: $foo"
done <<< "$(echo -e "$lines")"

You can get rid of the rather ugly echo in the here-string above by expanding the backslash sequences immediately when assigning lines. The $'...' form of quoting can be used there:

lines=$'first line\nsecond line\nthird line'
while read line; do
    ...
done <<< "$lines"

Programmatically replace transparent regions in an image with white fill?

I'm not sure how to detect transparent pixel. I know if the Alpha is 0 it's completly transparent and if it's 255 it's opaque. I'm not sure if you should check for Alpha == 0 or Alpha != 255 ; if you can try it and give me a feedback that would be helpful.

From MSDN

The alpha component specifies the
transparency of the color: 0 is fully
transparent, and 255 is fully opaque.
Likewise, an A value of 255 represents
an opaque color. An A value from 1
through 254 represents a
semitransparent color. The color
becomes more opaque as A approaches
255.

    void  Foo(Bitmap image)
    {
        for (int y = 0; y < image.Height; ++y)
        {
            for (int x = 0; x < image.Width; ++x)
            {
                // not very sure about the condition.                   
                if (image.GetPixel(x, y).A != 255)
                {
                    image.SetPixel(x,y,Color.White);
                }
            }
        }

    }

Set variable in current shell from awk

$ echo "$var"

$ declare $( awk 'BEGIN{print "var=17"}' )
$ echo "$var"
17

Here's why you should use declare instead of eval:

$ eval $( awk 'BEGIN{print "echo \"removing all of your files, ha ha ha....\""}' )
removing all of your files, ha ha ha....

$ declare $( awk 'BEGIN{print "echo \"removing all of your files\""}' )
bash: declare: `"removing': not a valid identifier
bash: declare: `files"': not a valid identifier

Note in the first case that eval executes whatever string awk prints, which could accidentally be a very bad thing!

---

Related Topics