Why Does Unix While Read Not Read Last Line

Why does unix while read not read last line?

What's happening is, the read command fails when the input is not terminated with a newline. Since the newline character is missing at the end of your file, the read fails, so the last iteration of the while loop is skipped.

If you don't want to / cannot make sure that your input file has a newline at the end, you can group your cat with an echo to give the appearance of an input terminated by newline, for example like this:

{ cat hello; echo; } | while read a b c d; do
    echo $a,$b,$c,$d
done

or like this:

(cat hello; echo) | while read a b c d; do
    echo $a,$b,$c,$d
done

while read loop ignoring last line in file

If the last line of your file has no newline on the end, read will put its content into card -- but will then exit with a nonzero value. Because read has exited with a nonzero value in this case, the while loop will exit without going on to the statement that runs the regex at all.

The easiest fix is to correct the file.

Another approach you can take is to ignore the exit status of read when it actually populates its destination (and, while at it, to put $'\r' into IFS, such that read will ignore the extra characters in DOS newlines):

while card=; IFS=$' \t\r\n' read -r card || [[ $card ]]; do
    if [[ $card =~ ^[0-9]{11}$ ]]
    then
        echo "some sql statement with $card" >> temp.sql;
    else
        echo "Invalid card number in file: $card";
    fi
done <registered/members_08_14.csv

Last line of a file is not reading in shell script

This is due to missing line break in the last line of your input file.

You can use this loop to read everything:

while IFS= read -r line || [ -n "$line" ]; do
    echo "$line"
done < "$file"

For the last line without line break, read doesn't return a success hence [ -n "$line" ] check is done to make sure to print it when $line is not empty.

PS: If you don't mind changing your input file then use printf to append a newline using:

printf '\n' >> "$file"

And then read normally:

while IFS= read -r line; do
    echo "$line"
done < "$file"

script not reading last line of a file

Use

grep . "${FILE_NAME}" | while read LINE

or

while read LINE
do
....
done < <(grep . "${FILE_NAME}")

The grep is less sensible to line-ending, and you will get empty-line skip for a free... :)

Honestly, never tried windows, all above is OK for unix...

EDIT Explanation:

make the next file:

echo -n -e 'line\n\nanother\nno line ending here>' >file.txt

the file contains 4 lines (although the last "line" is not a "correct" one)

line

another
no line ending here>

Usual shell routines, as read or wc looking for line ending. Therefore,

$ wc -l file.txt 
         3 file.txt

When you grepping for '' (empty string) the grep returns every line where found the string, so

$ grep '' file.txt

prints

line

another
no line ending here>

When grep prints out the found lines - ensures than one `\n' exists at the end, so

$ grep '' file.txt | wc -l

returns

4

therefore, for these situations, is better to use grep with -c (count) and not wc.

$ grep -c '' file.txt
4

Now, the . dot. The dot mean any character. So, when you grepping for a ., you get all lines what contain at least one character. And therefore, it will skip all lines what doesn't contain any character = skips empty lines. So,

$ grep . file.txt
line
another
no line ending here>

again, with added line ending to the last line (and skipped the empty line). Remember, the (space) is character too, so when the line contains only one space it is NOT EMPTY. Counting non-empty lines

$ grep . file.txt | wc -l
      3

or faster

$ grep -c . file.txt
3

bash: iterating through txt file lines can't read last line

if youre in doubt about the last \n in the file, you can try:

while read p; do
echo $p
done < <(grep '' file.txt)

grep is not picky about the line endings ;)

you can use grep . file.txt for skipping empty lines...

Bash loop only read the last line

First of all, awk has the ability to loop through lines and the field separator can be a regex.

So, your script can be reduced to this optimized format:

awk -F'[;:]' '{print $3}' 1.tmp > 1.tmp2

This is the optimized format that you can use.

Having said that, you might want to know what was wrong in the your script.

while read line ; do echo $line | awk -F':' '{print $3}' > 1.tmp2 ; done < 1.tmp
                                                         ^ here

The > marked above is the redirection operator.
It writes the stdout of the command (awk in this case) to the file specified. It does not append, but overwrite.
So, in every iteration of the loop, the file is cleared and the output of the command is written to it. Hence it leaves only the last entry.

To fix that, you can use the append redirection: >>.

while read line ; do echo $line | awk -F':' '{print $3}' >> 1.tmp2 ; done < 1.tmp

Now, there is a caveat. What if the file is not originally empty? This loop will append to the file, without clearing the file first. To fix that, you can first clear the file with:

>1.tmp2; while read line ; do echo $line | awk -F':' '{print $3}' >> 1.tmp2 ; done < 1.tmp

However, if we are sure that all the stdout produced by the loop needs to go into the file, you can simply move the redirection out of the loop. That way, shell does not have to keep opening and closing the file descriptors.

while read line ; do echo $line | awk -F':' '{print $3}'; done < 1.tmp > 1.tmp2

Note that these options are unoptimized, but would still work. The optimized option would be to let awk itself do the line-by-line processing as mentioned in the first snippet in the answer.

How to use `while read` (Bash) to read the last line in a file if there’s no newline at the end of the file?

In your first example, I'm assuming you are reading from stdin. To do the same with the second code block, you just have to remove the redirection and echo $REPLY:

DONE=false
until $DONE ;do
read || DONE=true
echo $REPLY
done

bash while loop drops last line of text file

Note:

21545a21548
> yheebash-3.00$
      ^---- no line break

Your file doesn't terminate with a line break.

---

Related Topics