While Using Printf How to Escape Special Characters in Shell Script

While using printf how to escape special characters in shell script?

You can use $' ' to enclose the newlines and tab characters, then a plain echo will suffice:

#!/bin/bash 

get_lines() {    
   local string
   string+='The path to K:\Users\ca, this is good'
   string+=$'\n'
   string+='The second line'
   string+=$'\t'
   string+='123'
   string+=$'\n'
   string+='It also has to be 100% nice than %99'

   echo "$string"
}

get_lines

I have also made a couple of other minor changes to your script. As well as making your FUNCTION_NAME lowercase, I have also used the more widely compatible function syntax. In this case, there's not a great deal of advantage (as $' ' strings are a bash extension anyway) but there's no reason to use the function func() syntax as far as I'm aware. Also, the scope of string may as well be local to the function in which it is used, so I changed that too.

Output:

The path to K:\Users\ca, this is good
The second line 123
It also has to be 100% nice than %99

Add escapes to special characters in a string (as a function)

Before I attempt to answer, I have a some warnings. I'm not sure what the actual goal is here, so depending on what that is, there are several potential problems.

First, it's impossible in general to reconstruct how a string was quoted/escaped on the command line, because there are many different ways to express the same string in shell syntax. For example, all of the following commands pass exactly the same argument to echo, and therefore print exactly the same thing:

echo This\ is\ a\ special\ character\ string.\ \\\~\`\!\@\#\$\%\^\&\*\(\)\-\=\+\[\]\{\}\|\;\:\'\"\,\<\>\/\?
echo This\ is\ a\ special\ character\ string.\ \\~\`\!@#\$%^\&*\(\)-=+[]{}\|\;:\'\",\<\>/?
echo 'This is a special character string. \~`!@#$%^&*()-=+[]{}|;:'"'"'",<>/?'
echo $'This is a special character string. \~`!@#$%^&*()-=+[]{}|;:\'",<>/?'
...and many more

(Note: technically, the second of those might pass something different, if the current directory happens to contain one or more files with specific really weird names.)

Second, some versions of echo do some additional escape processing. Some do this only when passed the -e option. Some print "-e" if you try to pass the -e option. It's a mess.

Third, what needs to be escaped and how really depends on what you're going to use it for (and specifically, how it's going to be parsed). Different situations involve different parsing rules, and you have to add escapes appropriate for the specific processing that the output is going to be subject to. In my answer, I concentrated on reversing the specific escaping in your example.

My solution: you can use sed in a pipeline to add escapes before any of a list of characters, specified as a bracket expression. It's slightly tricky because "]" and "-" are delimiters in a bracket expression; the trick there is to specify "]" as the first character and "-" as the last, so they're not mistaken for their other meanings. Also, I'm going to write this as a single-quoted string, so the single-quote requires special handing. Like this:

sed 's/[][ \~`!@#$%^&*()=+{}|;:'"'"'",<>/?-]/\\&/g'

Or as a function:

addESC() { sed 's/[][ \~`!@#$%^&*()=+{}|;:'"'"'",<>/?-]/\\&/g'; }

Example:

$ echo This\ is\ a\ special\ character\ string.\ \\\~\`\!\@\#\$\%\^\&\*\(\)\-\=\+\[\]\{\}\|\;\:\'\"\,\<\>\/\? | addESC 
This\ is\ a\ special\ character\ string.\ \\\~\`\!\@\#\$\%\^\&\*\(\)\-\=\+\[\]\{\}\|\;\:\'\"\,\<\>\/\?

As for why your attempts didn't work: In the first, printf doesn't read from stdin, it expects arguments. The third tries to fix this with xargs, but xargs does its own quote/escape parsing and removal, which messes it up. In the second, <<< takes a string, not a command; to apply it to the output of a command, you'd use something like <<< "$(command)". Also, in all versions, bash printf's %q quotes and/or escapes specifically as needed for consumption by bash itself, which doesn't match the escaping in your example.

How to escape special characters when echoing variable in Bash script?

If you do want a trailing newline on the input to cryptsetup:

s='^VfAro@khm=Y)@5,^AyxPO[[$2jW#+Vg;Paj2ycIj8VUr5z1,}qR~JnK7r_0@$ov'
printf '%s\n' "$s" | cryptsetup luksOpen

If you don't want a trailing newline:

s='^VfAro@khm=Y)@5,^AyxPO[[$2jW#+Vg;Paj2ycIj8VUr5z1,}qR~JnK7r_0@$ov'
printf '%s' "$s" | cryptsetup luksOpen

In both cases, we don't use echo when we care about byte-for-byte perfect output.

How to use printf %q in bash?

Bear this in mind when using %q:

ARGUMENT is printed in a format that can be reused as shell input, escaping non-printable characters with the proposed POSIX $'' syntax.

Emphasis mine. printf is free to reformat the arguments any way it likes as long as the input can be reused in the shell. However this is not the reason your input looks the way it does.

In Bash the ' character is a string delimiter, which is how you tell bash "the following string contains special characters like spaces, and these special characters should not be parsed by Bash". The quotes do not get passed to the commands that get called. What the command sees is something like this:

Command:
  printf "%q" a 'b c'

Received args:
  printf::arg0:  printf
  printf::arg1:  %q
  printf::arg2:  a
  printf::arg3:  b c

Note that arg3 does not have the quotes surrounding it. Bash does not pass them on.

When printf prints the args out, it does not know that there were quotes around b c, so it does not print them. But it does know that the space between 'b' and 'c' is a special shell character, and puts the \ in front to escape it.

This is true for all bash functions/commands, so bear in mind that the same happens when you call print_augs too.

If you want to maintain quotes around your strings, you'll need to double quote them so that Bash doesn't parse them:

function print_augs2() {
  echo "$@" >> "${output_file}"
}

print_augs2 a "'b c'"

# Output: a 'b c'

Print a string with its special characters printed as literal escape sequences

You will need to create a search and replace pattern for each binary value you wish to replace. Something like this:

#!/bin/bash

esc() {
    # space char after //
    v=${1// /\\s}   
    # tab character after //
    v=${v// /\\t}
    echo $v
}
esc "hello world"
esc "hello  world"

This outputs

hello\sworld
hello\tworld

Which characters need to be escaped when using Bash?

There are two easy and safe rules which work not only in sh but also bash.

1. Put the whole string in single quotes

This works for all chars except single quote itself. To escape the single quote, close the quoting before it, insert the single quote, and re-open the quoting.

'I'\''m a s@fe $tring which ends in newline
'

sed command: sed -e "s/'/'\\\\''/g; 1s/^/'/; \$s/\$/'/"

2. Escape every char with a backslash

This works for all characters except newline. For newline characters use single or double quotes. Empty strings must still be handled - replace with ""

\I\'\m\ \a\ \s\@\f\e\ \$\t\r\i\n\g\ \w\h\i\c\h\ \e\n\d\s\ \i\n\ \n\e\w\l\i\n\e"
"

sed command: sed -e 's/./\\&/g; 1{$s/^$/""/}; 1!s/^/"/; $!s/$/"/'.

2b. More readable version of 2

There's an easy safe set of characters, like [a-zA-Z0-9,._+:@%/-], which can be left unescaped to keep it more readable

I\'m\ a\ s@fe\ \$tring\ which\ ends\ in\ newline"
"

sed command: LC_ALL=C sed -e 's/[^a-zA-Z0-9,._+@%/-]/\\&/g; 1{$s/^$/""/}; 1!s/^/"/; $!s/$/"/'.

---

Note that in a sed program, one can't know whether the last line of input ends with a newline byte (except when it's empty). That's why both above sed commands assume it does not. You can add a quoted newline manually.

Note that shell variables are only defined for text in the POSIX sense. Processing binary data is not defined. For the implementations that matter, binary works with the exception of NUL bytes (because variables are implemented with C strings, and meant to be used as C strings, namely program arguments), but you should switch to a "binary" locale such as latin1.

---

(You can easily validate the rules by reading the POSIX spec for sh. For bash, check the reference manual linked by @AustinPhillips)

Storing escape characters in unix variable

The problem is caused by backticks. Use $( ) instead, and it goes away:

var="*a<br>*b<br>*c"
var=$(printf '%s\n' "$var" | sed 's/\*/\\*/g')
printf '%s\n' "$var"

(Why is this problem caused by backticks? Because the only way to nest them is to escape the inner ones with backslashes, so they necessarily change how backslashes behave; whereas $( ), because it uses different starting and ending sigils, can be nested natively).

---

That said, if your shell is one (like bash) with ksh-inspired extensions, you don't need sed at all here, as the shell can perform simple string replacements natively via parameter expansion:

var="*a<br>*b<br>*c"
printf '%s\n' "${var//'*'/'\*'}"

---

For background on why this answer uses printf instead of echo, see Why is printf better than echo? at [unix.se], or the APPLICATION USAGE section of the POSIX specification for echo.

How to escape a value returned from a command that contains special symbol in the shell script

You're using printf wrong. Its first argument is a format string, which tells it how to print the actual data (which is in the rest of the arguments). In the format string, % indicates a format specifier, which tells it to insert one of the data arguments (and the stuff immediately after % tells it how to format that data). Use something like this:

printf 'PASSWORD=%s\n' "$(az keyvault secret show --name app-PASSWORD --vault-name "my-vault" --query "value";)" >>vars.env

or maybe treat the "PASSWORD=" as data rather than part of the format:

printf '%s\n' "PASSWORD=$(az keyvault secret show --name app-PASSWORD --vault-name "my-vault" --query "value";)" >>vars.env

In both of these, the %s in the format string means "insert the next piece of data as a plain string".

---

Related Topics