Environment Variable Substitution in Sed

Environment variable substitution in sed

Your two examples look identical, which makes problems hard to diagnose. Potential problems:

1. You may need double quotes, as in sed 's/xxx/'"$PWD"'/'

2. $PWD may contain a slash, in which case you need to find a character not contained in $PWD to use as a delimiter.

To nail both issues at once, perhaps

sed 's@xxx@'"$PWD"'@'

sed | replace value with environment variable containing double quotes

Using envsubst:

Assume you have a string STRING which you want to replace with the variable VARIABLE_STRING. Then I would suggest the following approach:

1. substitute all variables of interest with a proper shell variable name (include the $)
2. export all variables of interest
3. Use envsubst to perform the substitution

Here is an example:

$ export VARIABLE_STRING='foo "bar" qux'
$ echo "This is the string 'STRING'" | sed 's/\bSTRING\b/$VARIABLE_STRING/g' | envsubst '$VARIABLE_STRING'
This is the string 'foo "bar" qux'

You can adapt this to your liking. Please read [U&L] Replacing only specific variables with envsubst for further information.

Using GNU awk:

If you cannot make use of envsubst, you can write something in awk. GNU awk can access the environment using the ENVIRON array which is a map from variable name to variable value. Example:

awk 'BEGIN{print ENVIRON["HOME"]}' 

This will print the content of $HOME

So now you could something like this:

1. Create a map from string to replace into environment
2. Loop over the names:

This would look something like this:

awk 'BEGIN{ map["NAME1"]="ENVIRONMENT_NAME1"
            map["NAME2"]="ENVIRONMENT_NAME2"
            map["NAME3"]="ENVIRONMENT_NAME3" }
     { for(i in map) gsub("/<" i "/>",ENVIRON[map[i]]) }
     { print }' file

Similar to the sed in the first solution, I've added a way to ensure that the string NAME1FOO is not being replaced by ENVIRONMENT_NAME1FOO. This by adding word_boundaries /< and /> in the regex.

**Note: this method might create loops. Imagine you have an environment variable:

ENVIRONMENT_NAME1="NAME2"

Then the string NAME1 might be replaced with the content of ENVIRONMENT_NAME2. I say might because everything depends on the order of transversal in the for loop which is awk dependent.

Sed string replace using environment variable

This steals from the answer to How can I interpret variables on the fly in the shell script? -- this question isn't quite a duplicate of that one.

Provided that the variables are part of the environment:

export FRUIT="APPLE"
export TIMES="DAILY"

You can use the envsubst GNU tool to perform the variable replacement:

echo 'The $$FRUIT$$ is good for health. Everyone should eat $$FRUIT$$ $$TIMES$$.' | sed -E 's/\$\$([A-Z]+)\$\$/${\1}/g' | envsubst
The APPLE is good for health. Everyone should eat APPLE DAILY.

Note that you need to put a backslash before the $ character since this character has a meaning for sed.

sed substitution with Bash variables

Variables inside ' don't get substituted in Bash. To get string substitution (or interpolation, if you're familiar with Perl) you would need to change it to use double quotes " instead of the single quotes:

# Enclose the entire expression in double quotes
$ sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp

# Or, concatenate strings with only variables inside double quotes
# This would restrict expansion to the relevant portion
# and prevent accidental expansion for !, backticks, etc.
$ sed 's/draw('"$prev_number"';n_)/draw('"$number"';n_)/g' file.txt > tmp

# A variable cannot contain arbitrary characters
# See link in the further reading section for details
$ a='foo
bar'
$ echo 'baz' | sed 's/baz/'"$a"'/g'
sed: -e expression #1, char 9: unterminated `s' command

Further Reading:

• Difference between single and double quotes in Bash
• Is it possible to escape regex metacharacters reliably with sed
• Using different delimiters for sed substitute command
• Unless you need it in a different file you can use the -i flag to change the file in place

Replacing text in a file using sed with an environment variable

You may try this:

find . -type f -name "*.js" -exec sed -i "s~http://localhost:3010~$SERVER~g" {} +

So using double quotes so that shell can expand $SERVER and also used alternate delimiter ~ because / is part of your variable.

sed variable substitution using string

You have two choices here. You can either evaluate $subdir to build up a fixed sed command that you then set as your filter; or you can evaluate a variable in the shell fragment that git filter-branch will invoke.

To understand the latter, realize that your --index-filter string becomes an ordinary shell variable:

--index-filter)
    filter_index="$OPTARG"
;;

which is then passed to eval:

eval "$filter_index" < /dev/null ||
    die "index filter failed: $filter_index"

The eval means that the expression in $filter_index, set from your --index-filter argument, has access to all the shell variables and functions in the filter-branch script. Unfortunately, none of its private variables holds the expression you'd like—but you can access its environment variables, which means you can put the value into an environment variable. In other words, you could supply subdir=<whatever> as an environment to your original expression.

In any case, as bk2204 answered, you need to remove the -i option. Besides that, some versions of sed don't accept \t as a tab character (presumably yours does, just be aware of this).

To expand the variable earlier, just do that. For instance:

... --index-filter \
    'git ls-files -s | sed "s-\t\"*-&'$folder'/-" | ...

(I've removed the -i here myself). Note how this exits single quotes, expands $folder, then re-enters single quotes. If $folder might contain whitespace, be sure to use double quotes while expanding it here:

... --index-filter \
    'git ls-files -s | sed "s-\t\"*-&'"$folder"'/-" | ...

The nesting of quotes here is pretty tricky: the stuff inside the single quotes is all one big string, provided as the argument that sets the variable $filter_index inside the filter-branch script. The eval runs it through a second pass of evaluation, breaking up into the pipeline (git ls-files, piped to sed, piped to git update-index) and running the various multiple commands, all of which have their stdin redirected to /dev/null.

Sed replace value in index.html by environment variable with slashes keeping enclosing text intact

You can use

sed -i 's/\(window\.API_URL =\)[^<]*/\1 "'"${MY_API_URL//\//\\\/}"'";/' index.html

See the online demo.

Details:

• \(window\.API_URL =\)[^<]* - a POSIX BRE pattern that matches and captures into Group 1 window.API_URL = text and then matches any zero or more chars other than <
• \1 "'"${MY_API_URL//\//\\\/}"'"; - a replacement that puts back the value captured into Group 1 (\1) and then adds a space, ", the MY_API_URL value and then "; string.

The ${MY_API_URL//\//\\\/} is an example of variable expansion where a / is replaced with \/ (thanks to // after the variable name, all occurrences are replaced).

Using Sed to expand environment variables inside files

Consider your trial version:

cat test | sed -e "s/\(\${[A-Z]*}\)/`eval "echo '\1'"`/" > outputfile

The reason this doesn't work is because it requires prescience on the part of the shell. The sed script is generated before any pattern is matched by sed, so the shell cannot do that job for you.

I've done this a couple of ways in the past. Normally, I've had a list of known variables and their values, and I've done the substitution from that list:

for var in PATH VARIABLE USERNAME
do
    echo 's%${'"$var"'}%'$(eval echo "\$$var")'%g'
done > sed.script

cat test | sed -f sed.script > outputfile

If you want to map variables arbitrarily, then you either need to deal with the whole environment (instead of the fixed list of variable names, use the output from env, appropriately edited), or use Perl or Python instead.

Note that if the value of an environment variable contains a slash in your version, you'd run into problems using the slash as the field separator in the s/// notation. I used the '%' since relatively few environment variables use that - but there are some found on some machines that do contain '%' characters and so a complete solution is trickier. You also need to worry about backslashes in the value. You probably have to use something like '$(eval echo "\$$var" | sed 's/[\%]/\\&/g')' to escape the backslashes and percent symbols in the value of the environment variable. Final wrinkle: some versions of sed have (or had) a limited capacity for the script size - older versions of HP-UX had a limit of about 100. I'm not sure whether that is still an issue, but it was as recently as 5 years ago.

The simple-minded adaptation of the original script reads:

env |
sed 's/=.*//' |
while read var
do
    echo 's%${'"$var"'}%'$(eval echo "\$$var" | sed 's/[\%]/\\&/g')'%g'
done > sed.script

cat test | sed -f sed.script > outputfile

However, a better solution uses the fact that you already have the values in the output from env, so we can write:

env |
sed 's/[\%]/\\&/g;s/\([^=]*\)=\(.*\)/s%${\1}%\2%/' > sed.script
cat test | sed -f sed.script > outputfile

This is altogether safer because the shell never evaluates anything that should not be evaluated - you have to be so careful with shell metacharacters in variable values. This version can only possibly run into any trouble if some output from env is malformed, I think.

Beware - writing sed scripts with sed is an esoteric occupation, but one that illustrates the power of good tools.

All these examples are remiss in not cleaning up the temporary file(s).

---

Related Topics