In Bash Tee Is Making Function Variables Local, How to Escape This

In bash tee is making function variables local, how do I escape this?

This is a variant of BashFAQ #24.

var 2>&1 | tee $LOGV

...like any shell pipeline, has the option to run the function var inside a subprocess -- and, in practice, behaves this way in bash. (The POSIX sh specification leaves the details of which pipeline components, if any, run inside the parent shell undefined).

Avoiding this is as simple as not using a pipeline.

var > >(tee "$LOGV") 2>&1

...uses process substitution (a ksh extension adopted by bash, not present in POSIX sh) to represent the tee subprocess through a filename (in the form /dev/fd/## on modern Linux) which output can be redirected to without moving the function into a pipeline.

If you want to ensure that tee exits before other commands run, use a lock:

#!/bin/bash
logv=/tmp/log

collect_var() {
        echo "value for var:"
        read -re var
}
collect_var > >(logv="$logv" flock "$logv" -c 'exec tee "$logv"') 2>&1
flock "$logv" -c true # wait for tee to exit

echo "This is var: $var"

Incidentally, if you want to run multiple commands with their output being piped in this way, you should invoke the tee only once, and feed into it as appropriate:

#!/bin/bash
logv=/tmp/log
collect_var() { echo "value for var:"; read -re var; }

exec 3> >(logv="$logv" flock "$logv" -c 'exec tee "$logv"') # open output to log
collect_var >&3 2>&3         # run function, sending stdout/stderr to log
echo "This is var: $var" >&3 # ...and optionally run other commands the same way
exec 3>&-                    # close output
flock "$logv" -c true        # ...and wait for tee to finish flushing and exit.

Using variables inside a bash heredoc

In answer to your first question, there's no parameter substitution because you've put the delimiter in quotes - the bash manual says:

The format of here-documents is:
      <<[-]word
              here-document
      delimiter
No parameter expansion, command substitution, arithmetic expansion, or
pathname expansion is performed on word. If any characters in word are
quoted, the delimiter is the result of quote removal on word, and the
lines in the here-document are not expanded. If word is unquoted, all
lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. [...]

If you change your first example to use <<EOF instead of << "EOF" you'll find that it works.

In your second example, the shell invokes sudo only with the parameter cat, and the redirection applies to the output of sudo cat as the original user. It'll work if you try:

sudo sh -c "cat > /path/to/outfile" <<EOT
my text...
EOT

read stdin in function in bash script

If the question is How do I pass stdin to a bash function?, then the answer is:

Shellscript functions take stdin the ordinary way, as if they were commands or programs. :)

input.txt:

HELLO WORLD
HELLO BOB
NO MATCH

test.sh:

#!/bin/sh

myfunction() {
    grep HELLO
}

cat input.txt | myfunction

Output:

hobbes@metalbaby:~/scratch$ ./test.sh 
 HELLO WORLD 
 HELLO BOB

Note that command line arguments are ALSO handled in the ordinary way, like this:

test2.sh:

#!/bin/sh

myfunction() {
    grep "$1"
}

cat input.txt | myfunction BOB

Output:

hobbes@metalbaby:~/scratch/$ ./test2.sh 
 HELLO BOB

How to cat EOF a file containing code?

You only need a minimal change; single-quote the here-document delimiter after <<.

cat <<'EOF' >> brightup.sh

or equivalently backslash-escape it:

cat <<\EOF >>brightup.sh

Without quoting, the here document will undergo variable substitution, backticks will be evaluated, etc, like you discovered.

If you need to expand some, but not all, values, you need to individually escape the ones you want to prevent.

cat <<EOF >>brightup.sh
#!/bin/sh
# Created on $(date # : <<-- this will be evaluated before cat;)
echo "\$HOME will not be evaluated because it is backslash-escaped"
EOF

will produce

#!/bin/sh
# Created on Fri Feb 16 11:00:18 UTC 2018
echo "$HOME will not be evaluated because it is backslash-escaped"

As suggested by @fedorqui, here is the relevant section from man bash:

Here Documents
This type of redirection instructs the shell to read input from the
current source until a line containing only delimiter (with no
trailing blanks) is seen. All of the lines read up to that point are
then used as the standard input for a command.
The format of here-documents is:
      <<[-]word
              here-document
      delimiter
No parameter expansion, command substitution, arithmetic expansion,
or pathname expansion is performed on word. If any characters in word
are quoted, the delimiter is the result of quote removal on word, and
the lines in the here-document are not expanded. If word is
unquoted, all lines of the here-document are subjected to parameter
expansion, command substitution, and arithmetic expansion. In the
latter case, the character sequence \<newline> is ignored, and \
must be used to quote the characters \, $, and `.

How to store standard error in a variable

It would be neater to capture the error file thus:

ERROR=$(</tmp/Error)

The shell recognizes this and doesn't have to run 'cat' to get the data.

The bigger question is hard. I don't think there's an easy way to do it. You'd have to build the entire pipeline into the sub-shell, eventually sending its final standard output to a file, so that you can redirect the errors to standard output.

ERROR=$( { ./useless.sh | sed s/Output/Useless/ > outfile; } 2>&1 )

Note that the semi-colon is needed (in classic shells - Bourne, Korn - for sure; probably in Bash too). The '{}' does I/O redirection over the enclosed commands. As written, it would capture errors from sed too.

WARNING: Formally untested code - use at own risk.

How to quotes in bash function parameters?

The reason this happens is because bash interprets the arguments, as you thought. The quotes simply aren't there any more when it calls the function, so this isn't possible. It worked in DOS because programs could interpret the command line themselves, not that it helps you!

bash functions: enclosing the body in braces vs. parentheses

Why are braces used by default to enclose the function body instead of parentheses?

The body of a function can be any compound command. This is typically { list; }, but three other forms of compound commands are technically allowed: (list), ((expression)), and [[ expression ]].

C and languages in the C family like C++, Java, C#, and JavaScript all use curly braces to delimit function bodies. Curly braces are the most natural syntax for programmers familiar with those languages.

Are there other major downsides (*) to using parentheses instead of braces (which might explain why braces seem to be preferred)?

Yes. There are numerous things you can't do from a sub-shell, including:

Change global variables. Variables changes will not propagate to the parent shell.
Exit the script. An exit statement will exit only the sub-shell.

Starting a sub-shell can also be a serious performance hit. You're launching a new process each time you call the function.

You might also get weird behavior if your script is killed. The signals the parent and child shells receive will change. It's a subtle effect but if you have trap handlers or you kill your script those parts not work the way you want.

When shall I use curly braces to enclose the function body, and when is it advisable to switch to parentheses?

I would advise you to always use curly braces. If you want an explicit sub-shell, then add a set of parentheses inside the curly braces. Using just parentheses is highly unusual syntax and would confuse many people reading your script.

foo() {
   (
       subshell commands;
   )
}

Pipe output and capture exit status in Bash

There is an internal Bash variable called $PIPESTATUS; it’s an array that holds the exit status of each command in your last foreground pipeline of commands.

<command> | tee out.txt ; test ${PIPESTATUS[0]} -eq 0

Or another alternative which also works with other shells (like zsh) would be to enable pipefail:

set -o pipefail
...

The first option does not work with zsh due to a little bit different syntax.

How to replace ${} placeholders in a text file?

Update

Here is a solution from yottatsa on a similar question that only does replacement for variables like $VAR or ${VAR}, and is a brief one-liner

i=32 word=foo envsubst < template.txt

Of course if i and word are in your environment, then it is just

envsubst < template.txt

On my Mac it looks like it was installed as part of gettext and from MacGPG2

Old Answer

Here is an improvement to the solution from mogsie on a similar question, my solution does not require you to escale double quotes, mogsie's does, but his is a one liner!

eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null

The power on these two solutions is that you only get a few types of shell expansions that don't occur normally $((...)), `...`, and $(...), though backslash is an escape character here, but you don't have to worry that the parsing has a bug, and it does multiple lines just fine.

Echo newline in Bash prints literal \n

Use printf instead:

printf "hello\nworld\n"

printf behaves more consistently across different environments than echo.