Can't Use a Variable Out of While and Pipe in Bash

Can't use a variable out of while and pipe in bash

The reason for this behaviour is that a while loop runs in a subshell when it's part of a pipeline. For the while loop above, a new subshell with its own copy of the variable var is created.

See this article for possible workarounds: I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?.

A variable modified inside a while loop is not remembered

echo -e $lines | while read line 
    ...
done

The while loop is executed in a subshell. So any changes you do to the variable will not be available once the subshell exits.

Instead you can use a here string to re-write the while loop to be in the main shell process; only echo -e $lines will run in a subshell:

while read line
do
    if [[ "$line" == "second line" ]]
    then
        foo=2
        echo "Variable \$foo updated to $foo inside if inside while loop"
    fi
    echo "Value of \$foo in while loop body: $foo"
done <<< "$(echo -e "$lines")"

You can get rid of the rather ugly echo in the here-string above by expanding the backslash sequences immediately when assigning lines. The $'...' form of quoting can be used there:

lines=$'first line\nsecond line\nthird line'
while read line; do
    ...
done <<< "$lines"

How to pipe input to a Bash while loop and preserve variables after loop ends

The correct notation for Process Substitution is:

while read i; do echo $i; done < <(echo "$FILECONTENT")

The last value of i assigned in the loop is then available when the loop terminates.
An alternative is:

echo $FILECONTENT | 
{
while read i; do echo $i; done
...do other things using $i here...
}

The braces are an I/O grouping operation and do not themselves create a subshell. In this context, they are part of a pipeline and are therefore run as a subshell, but it is because of the |, not the { ... }. You mention this in the question. AFAIK, you can do a return from within these inside a function.

---

Bash also provides the shopt builtin and one of its many options is:

lastpipe

If set, and job control is not active, the shell runs the last command of a pipeline not executed in the background in the current shell environment.

Thus, using something like this in a script makes the modfied sum available after the loop:

FILECONTENT="12 Name
13 Number
14 Information"
shopt -s lastpipe   # Comment this out to see the alternative behaviour
sum=0
echo "$FILECONTENT" |
while read number name; do ((sum+=$number)); done
echo $sum

Doing this at the command line usually runs foul of 'job control is not active' (that is, at the command line, job control is active). Testing this without using a script failed.

Also, as noted by Gareth Rees in his answer, you can sometimes use a here string:

while read i; do echo $i; done <<< "$FILECONTENT"

This doesn't require shopt; you may be able to save a process using it.

Shell variables set inside while loop not visible outside of it

When you pipe into a while loop in Bash, it creates a subshell. When the subshell exits, all variables return to their previous values (which may be null or unset). This can be prevented by using process substitution.

LONGEST_CNT=0
while read -r line
do
    line_length=${#line}
    if (( line_length > LONGEST_CNT ))
    then
        LONGEST_CNT=$line_length
        LONGEST_STR=$line
    fi
done < <(find samples/ )    # process substitution

echo $LONGEST_CNT : $LONGEST_STR

Variable value not getting preserved when piping to another program

Your guess is correct. Pipes spawn subshells and variables changed in a subshell don't change the parent's variables.

You can switch the pipe to process substitution, which will run column in a child process but leave directory_chooser in the parent.

directory_chooser "$1" > >(column -t -s $'\t')

Command not found piping a variable to cut when output stored in a variable

You need to pass an input string to the shell command using a pipeline in which case cut or any standard shell commands, reads from stdin and acts on it. Some of the ways you can do this are use a pipe-line

dir2=$(echo "$MY_DIR" | cut -d'/' -f-4)

(or) use a here-string which is a shell built-in instead of launching a external shell process

dir2=$(cut -d'/' -f-4 <<< "$MY_DIR")

Propagate value of variable to outside of the loop

The problem is the pipe, not the loop. Try it this way

let i=0
arr=()
_constr=

while read -r line ; do
    arr=("${line}")
    let i=i+1
    _constr+="${arr[2]} "
done < <(dpkg --list | grep linux-image | grep 'ii  ')

echo "$i"
echo "${_constr}"

Pipes are executed in a subshell, as noted by Blagovest in his comment. Using process substitution instead (this is the < <(commands) syntax) keeps everything in the same process, so changes to global variables are possible.

Incidentally, your pipeline could be improved as well

dpkg --list | grep '^ii.*linux-image'

One less invocation of grep to worry about.

Variables getting reset after the while read loop that reads from a pipeline

I ran into this problem yesterday.

The trouble is that you're doing find $loc -name "*.bsp" | while read. Because this involves a pipe, the while read loop can't actually be running in the same bash process as the rest of your script; bash has to spawn off a subprocess so that it can connect the the stdout of find to the stdin of the while loop.

This is all very clever, but it means that any variables set in the loop can't be seen after the loop, which totally defeated the whole purpose of the while loop I was writing.

You can either try to feed input to the loop without using a pipe, or get output from the loop without using variables. I ended up with a horrifying abomination involving both writing to a temporary file AND wrapping the whole loop in $(...), like so:

var="$(producer | while read line; do
    ...
    echo "${something}"
done)"

Which got me var set to all the things that had been echoed from the loop. I probably messed up the syntax of that example; I don't have the code I wrote handy at the moment.

Read values into a shell variable from a pipe

Use

IFS= read var << EOF
$(foo)
EOF

You can trick read into accepting from a pipe like this:

echo "hello world" | { read test; echo test=$test; }

or even write a function like this:

read_from_pipe() { read "$@" <&0; }

But there's no point - your variable assignments may not last! A pipeline may spawn a subshell, where the environment is inherited by value, not by reference. This is why read doesn't bother with input from a pipe - it's undefined.

FYI, http://www.etalabs.net/sh_tricks.html is a nifty collection of the cruft necessary to fight the oddities and incompatibilities of bourne shells, sh.

---

Related Topics