In Bash, How to Expand Variables Twice in Double Quotes

In Bash, is there a way to expand variables twice in double quotes?

You cannot safely evaluate expansions twice when handling runtime data.

There are means to do re-evaluation, but they require trusting your data -- in the NSA system design sense of the word: "A trusted component is one that can break your system when it fails".

See BashFAQ #48 for a detailed discussion. Keep in mind that if you could be logging filenames, that any character except NUL can be present in a UNIX filename. $(rm -rf ~)'$(rm -rf ~)'.txt is a legal name. * is a legal name.

Consider a different approach:

#!/usr/bin/env bash

trace() { echo "${FUNCNAME[1]}:${BASH_LINENO[0]}: $*" >&2; }

foo() {
        bar=baz
        trace "I just set bar to $bar"
}

foo

...which, when run with bash 4.4.19(1)-release, emits:

foo:7: I just set bar to baz

---

Note the use of ${BASH_LINENO[0]} and ${FUNCNAME[1]}; this is because BASH_LINENO is defined as follows:

An array variable whose members are the line numbers in source files where each corresponding member of FUNCNAME was invoked.

Thus, FUNCNAME[0] is trace, whereas FUNCNAME[1] is foo; whereas BASH_LINENO[0] is the line from which trace was called -- a line which is inside the function foo.

bash quotes in variable treated different when expanded to command

The important thing to keep in mind is that quotes are only removed once, when the command line is originally parsed. A quote which is inserted into the command line as a result of parameter substitution ($foo) or command substitution ($(cmd args)) is not treated as a special character. [Note 1]

That seems different from whitespace and glob metacharacters. Word splitting and pathname expansion happen after parameter/command substitution (unless the substitution occurs inside quotes). [Note 2]

The consequence is that it is almost impossible to create a bash variable $args such that

cmd $args

If $args contains quotes, they are not removed. Words inside $args are delimited by sequences of whitespace, not single whitespace characters.

The only way to do it is to set $IFS to include some non-whitespace character; that character can then be used inside $args as a single-character delimiter. However, there is no way to quote a character inside a value, so once you do that, the character you chose cannot be used other than as a delimiter. This is not usually very satisfactory.

There is a solution, though: bash arrays.

If you make $args into an array variable, then you can expand it with the repeated-quote syntax:

cmd "${args[@]}"

which produces exactly one word per element of $args, and suppresses word-splitting and pathname expansion on those words, so they end up as literals.

So, for example:

actions=(--tags all:)
actions+=(--chapters '')
mkvpropedit "$1" "${actions[@]}"

will probably do what you want. So would:

args=("$1")
args+=(--tags)
args+=(all:)
args+=(--chapters)
args+=('')
mkvpropedit "${args[@]}"

and so would

command=(mkvpropedit "$1" --tags all: --chapters '')
"${command[@]}"

I hope that's semi-clear.

man bash (or the online version) contains a blow-by-blow account of how bash assembles commands, starting at the section "EXPANSION". It's worth reading for a full explanation.

---

Notes:

1. This doesn't apply to eval or commands like bash -c which evaluate their argument again after command line processing. But that's because command-line processing happens twice.

2. Word splitting is not the same as "dividing the command into words", which happens when the command is parsed. For one thing, word-splitting uses as separator characters the value of $IFS, whereas command-line parsing uses whitespace. But neither of these are done inside quotes, so they are similar in that respect. In any case, words are split in one way or another both before and after parameter substitution.

Expansion of bash variable in multiple quotes?

If you have double quotes around your whole command, you can insert single quotes without any trouble but need to escape double quotes.

For example:

$ watcher=jsimmons
$ echo "'\"$watcher\"'"
'"jsimmons"'

Multi-line, double quoted string triggers history expansion on subsequent single-quoted commands it gets piped to

It is not a bug.

After asking this in the bash-bugs mailing list, I got the following answer:

History expansion is explicitly line-oriented.

It doesn't know about shell state, especially shell quoting state,
that spans lines.

It does know about vaguely shell-like quoting that's common across a
large set of Unix utilities -- since the history and readline
libraries are used outside the shell -- and that a double quote
introduces a quoted string in which single quotes are not significant
and don't inhibit history expansion.

Bash expand variable in a variable

Using eval is the classic solution, but bash has a better (more easily controlled, less blunderbuss-like) solution:

• ${!colour}

The Bash (4.1) reference manual says:

If the first character of parameter is an exclamation point (!), a level of variable indirection
is introduced. Bash uses the value of the variable formed from the rest of parameter as
the name of the variable; this variable is then expanded and that value is used in the rest
of the substitution, rather than the value of parameter itself. This is known as indirect
expansion.

For example:

$ Green=$'\033[32;m'
$ echo "$Green" | odx
0x0000: 1B 5B 33 32 3B 6D 0A                              .[32;m.
0x0007:
$ colour=Green
$ echo $colour
Green
$ echo ${!colour} | odx
0x0000: 1B 5B 33 32 3B 6D 0A                              .[32;m.
0x0007:
$

(The odx command is very non-standard but simply dumps its data in a hex format with printable characters shown on the right. Since the plain echo didn't show anything and I needed to see what was being echoed, I used an old friend I wrote about 24 years ago.)

Use variable inside double quotes inside single quotes in sed

Use double quotes to quote the sed commands as

sed -i "s/\"${FOO}\"/bar/m" file.txt

Test:

$ echo $b
hello
$ echo \"hello\" | sed "s/\"$b\"/asdf/"
asdf

Bash quoting of current path (pwd)

Replace single quotes by double quotes and replace quotes with backquotes around pwd:

mycommand | sed -E "
 s|mystuff|replacement| ;
 s|`pwd`|replacement| ;
"

Double quotes allow expansion of variables and backquoted commands.

---

Related Topics