How to Loop Over the Output of a Shell Command

How can I loop over the output of a shell command?

Never for loop over the results of a shell command if you want to process it line by line unless you are changing the value of the internal field separator $IFS to \n. This is because the lines will get subject of word splitting which leads to the actual results you are seeing. Meaning if you for example have a file like this:

foo bar
hello world

The following for loop

for i in $(cat file); do
echo "$i"
done

gives you:

foo
bar
hello
world

Even if you use IFS='\n' the lines might still get subject of Filename expansion


I recommend to use while + read instead because read reads line by line.

Furthermore I would use pgrep if you are searching for pids belonging to a certain binary. However, since python might appear as different binaries, like python2.7 or python3.4 I suggest to pass -f to pgrep which makes it search the whole command line rather than just searching for binaries called python. But this will also find processes which have been started like cat foo.py. You have been warned! At the end you can refine the regex passed to pgrep like you wish.

Example:

pgrep -f python | while read -r pid ; do
echo "$pid"
done

or if you also want the process name:

pgrep -af python | while read -r line ; do
echo "$line"
done

If you want the process name and the pid in separate variables:

pgrep -af python | while read -r pid cmd ; do
echo "pid: $pid, cmd: $cmd"
done

You see, read offers a flexible and stable way to process the output of a command line-by-line.


Btw, if you prefer your ps .. | grep command line over pgrep use the following loop:

ps -ewo pid,etime,cmd | grep python | grep -v grep | grep -v sh \
| while read -r pid etime cmd ; do
echo "$pid $cmd $etime"
done

Note how I changed the order of etime and cmd. Thus to be able to read cmd, which can contain whitespace, into a single variable. This works because read will break down the line into variables, as many times as you specified variables. The remaining part of the line - possibly including whitespace - will get assigned to the last variable which has been specified in the command line.

Looping through the content of a file in Bash

One way to do it is:

while read p; do
echo "$p"
done <peptides.txt

As pointed out in the comments, this has the side effects of trimming leading whitespace, interpreting backslash sequences, and skipping the last line if it's missing a terminating linefeed. If these are concerns, you can do:

while IFS="" read -r p || [ -n "$p" ]
do
printf '%s\n' "$p"
done < peptides.txt

Exceptionally, if the loop body may read from standard input, you can open the file using a different file descriptor:

while read -u 10 p; do
...
done 10<peptides.txt

Here, 10 is just an arbitrary number (different from 0, 1, 2).

How to loop through a list in shell?

Use a bash while-loop, the loop can be done over a command or an input file.

while IFS= read -r string
do
some_stuff to do
done < <(command_that_produces_string)

With an example, I have a sample file with contents as

$ cat file
My
name
is not
relevant
here

I have modified the script to echo the line as it reads through the file

$ cat script.sh
#!/bin/bash
while IFS= read -r string
do
echo "$string"
done < file

produces an o/p when run as ./script.sh

My
name
is not
relevant
here

The same can also be done over a bash-command, where we adopt process-substitution (<()) to run the command on the sub-shell.

#!/bin/bash

while IFS= read -r -d '' file; do
echo "$file"
done < <(find . -maxdepth 1 -mindepth 1 -name "*.txt" -type f -print0)

The above simple find lists all files from the current directory (including ones with spaces/special-characters). Here, the output of find command is fed to stdin which is parsed by while-loop.

ansible loop over shell command output

I needed to do this with_indexed_items: "{{ vcs.stdout.split('\n')}}"

How to process each output line in a loop?

One of the easy ways is not to store the output in a variable, but directly iterate over it with a while/read loop.

Something like:

grep xyz abc.txt | while read -r line ; do
echo "Processing $line"
# your code goes here
done

There are variations on this scheme depending on exactly what you're after.

If you need to change variables inside the loop (and have that change be visible outside of it), you can use process substitution as stated in fedorqui's answer:

while read -r line ; do
echo "Processing $line"
# your code goes here
done < <(grep xyz abc.txt)


Related Topics



Leave a reply



Submit