Looping Through Lines in a File in Bash, Without Using Stdin

Looping through lines in a file in bash, without using stdin

You must open as a different file descriptor

while read p <&3; do
echo "$p"
echo 'Hit enter for the next one'
read x
done 3< list.txt

Update: Just ignore the lengthy discussion in the comments below. It has nothing to do with the question or this answer.

Looping through the content of a file in Bash

One way to do it is:

while read p; do
echo "$p"
done <peptides.txt

As pointed out in the comments, this has the side effects of trimming leading whitespace, interpreting backslash sequences, and skipping the last line if it's missing a terminating linefeed. If these are concerns, you can do:

while IFS="" read -r p || [ -n "$p" ]
do
printf '%s\n' "$p"
done < peptides.txt

Exceptionally, if the loop body may read from standard input, you can open the file using a different file descriptor:

while read -u 10 p; do
...
done 10<peptides.txt

Here, 10 is just an arbitrary number (different from 0, 1, 2).

Looping through a file but ignoring the first two lines in Bash

Given the small number of lines (2) to be ignored, I suggest the following:

{
read; read
while IFS="" read -r p || [ -n "$p" ] ; do
...
done
} < diff.txt

read; read reads the first two lines from stdin (diff.txt).

How do I iterate over each line in a file with Bash?

Use cat for concatenating or displaying. No need for it here.

file="/path/to/file"
while read line; do
echo "${line}"
done < "${file}"

bash script won't loop over lines in file when certain command executed

The problem was the same as:

I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!

so the solution was to redirect stdin:

rosbag play -r 10 $bag </dev/null

reading a file line by line using the shell

while IFS= read -r line < input_file
do
echo "$line"
done

In this scenario, you are re-initiating the read of input_file with every iteration of the loop. As read reads line by line, each re-initialisation will print the first line. As you never progress past this first line, it will print the first line only indefinitely

while IFS= read -r line
do
echo "$line"
done < input_file

In this scenario, the input_file is redirected in to the while loop and then the read line takes place with each line of the redirected output until it gets to the end of the output and then end of the file.

An alternate way of processing the file is to cat file_input through to the while loop:

cat input_file | while IFS= read -r line
do
echo "$line"
done

This works in the same way as the previous section but is less efficient due to the pipe. The flow of the output runs from left to right as opposed to right to left. This is often easier to understand. The output of cat input_file is fed (piped) into the while loop and each line is read till the end of the output/file.

Using bash for loop to iterate through file

The for loop takes its iteration values from its in clause or from the content of $@; it does not read from stdin.

In your last example, the entire command's stdin is being redirected from a.txt and it produces no output because for doesn't read from stdin and its command block's commands don't either.

The read command, however, does read from stdin and that's why the while read arg; loop works.

Credit to Glenn for explaining the meaning of a for command without an in clause:

The value of $@ is empty and that is why your for loop produces no output. Redirecting a command's stdin from a file does not set $@. If you had placed that for loop in a script and executed it with some arguments, then your for loop would have printed the command line's arguments instead of the content of a.txt.

Read a file line by line assigning the value to a variable

The following reads a file passed as an argument line by line:

while IFS= read -r line; do
echo "Text read from file: $line"
done < my_filename.txt

This is the standard form for reading lines from a file in a loop. Explanation:

  • IFS= (or IFS='') prevents leading/trailing whitespace from being trimmed.
  • -r prevents backslash escapes from being interpreted.

Or you can put it in a bash file helper script, example contents:

#!/bin/bash
while IFS= read -r line; do
echo "Text read from file: $line"
done < "$1"

If the above is saved to a script with filename readfile, it can be run as follows:

chmod +x readfile
./readfile filename.txt

If the file isn’t a standard POSIX text file (= not terminated by a newline character), the loop can be modified to handle trailing partial lines:

while IFS= read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
done < "$1"

Here, || [[ -n $line ]] prevents the last line from being ignored if it doesn't end with a \n (since read returns a non-zero exit code when it encounters EOF).

If the commands inside the loop also read from standard input, the file descriptor used by read can be chanced to something else (avoid the standard file descriptors), e.g.:

while IFS= read -r -u3 line; do
echo "Text read from file: $line"
done 3< "$1"

(Non-Bash shells might not know read -u3; use read <&3 instead.)



Related Topics



Leave a reply



Submit