How to Substitute Shell Variables in Complex Text Files

How to substitute shell variables in complex text files

Looking, it turns out on my system there is an envsubst command which is part of the gettext-base package.

So, this makes it easy:

envsubst < "source.txt" > "destination.txt"

Note if you want to use the same file for both, you'll have to use something like moreutil's sponge, as suggested by Johnny Utahh: envsubst < "source.txt" | sponge "source.txt". (Because the shell redirect will otherwise empty the file before its read.)

How to replace ${} placeholders in a text file?

Sed!

Given template.txt:

The number is ${i}
The word is ${word}

we just have to say:

sed -e "s/\${i}/1/" -e "s/\${word}/dog/" template.txt

Thanks to Jonathan Leffler for the tip to pass multiple -e arguments to the same sed invocation.

Complex string with special characters substitutions

some meta-characters like \ [ $ must be quoted in single quotes (which is not possible in your case) or escaped (with backslash, which must escaped with itself too)

you can use string manipulation for replacing

run this function over your strings

escape() {
  local var="$1"

  var="${var//\\/\\\\}" # \
  var="${var//\[/\\\[}" # [
  var="${var//\$/\\\$}" # $
  printf "%s" "$var"

  return 0
}

VarA=$(escape "$VarA")
VarB=$(escape "$VarB")

Environment variable substitution in sed

Your two examples look identical, which makes problems hard to diagnose. Potential problems:

1. You may need double quotes, as in sed 's/xxx/'"$PWD"'/'

2. $PWD may contain a slash, in which case you need to find a character not contained in $PWD to use as a delimiter.

To nail both issues at once, perhaps

sed 's@xxx@'"$PWD"'@'

How do I parse a file and substitute the environment variables with values in Python

string.Template

>>> from string import Template
>>> s = Template('$who likes $what')
>>> s.substitute(who='tim', what='kung pao')
'tim likes kung pao'

Find and Replace Inside a Text File from a Bash Command

The easiest way is to use sed (or perl):

sed -i -e 's/abc/XYZ/g' /tmp/file.txt

Which will invoke sed to do an in-place edit due to the -i option. This can be called from bash.

If you really really want to use just bash, then the following can work:

while IFS='' read -r a; do
    echo "${a//abc/XYZ}"
done < /tmp/file.txt > /tmp/file.txt.t
mv /tmp/file.txt{.t,}

This loops over each line, doing a substitution, and writing to a temporary file (don't want to clobber the input). The move at the end just moves temporary to the original name. (For robustness and security, the temporary file name should not be static or predictable, but let's not go there.)

For Mac users:

sed -i '' 's/abc/XYZ/g' /tmp/file.txt

(See the comment below why)

Makefile: Reuse environment variables

If the .ONESHELL special target appears anywhere in the makefile then all recipe lines for each target will be provided to a single invocation of the shell. Newlines between recipe lines will be preserved.

.ONESHELL:
deploy:
  export $$(this-script-generate-key-values.sh)
  echo $${VAR1} 
  echo $${VAR2} 

---

Related Topics