Shell Script Printing Contents of Variable Containing Output of a Command Removes Newline Characters

Shell script printing contents of variable containing output of a command removes newline characters

If you want to preserve the newlines, enclose the variable in double quotes:

echo "$stuff"

When you write it without the double quotes, the shell expands $stuff into a space-separated list of words (where 'words' are sequences of non-space characters, and the space characters are blanks and tabs and newlines; upon experimentation, it seems that form feeds, carriage returns and back-spaces are not counted as space).

---

Demonstrating interpretation of control characters as white space. ASCII 8 is backspace, 9 is tab, 10 is new line (LF), 11 is vertical tab, 12 is form feed, 13 is carriage return. The first command generates a sequence of characters separated by the various control characters. The second command echoes with the result with the original characters preserved - see the hex dump. The third command echoes the result with the shell splitting the words; you can see that the tab and newline were replaced by blank (0x20).

$ x=$(./ascii 64 65 8 66 67 9 68 69 10 70 71 11 72 73 12 74 75 13 76 77)
$ echo "$x" | odx
0x0000: 40 41 08 42 43 09 44 45 0A 46 47 0B 48 49 0C 4A   @A.BC.DE.FG.HI.J
0x0010: 4B 0D 4C 4D 0A                                    K.LM.
0x0015:
$ echo  $x  | odx
0x0000: 40 41 08 42 43 20 44 45 20 46 47 0B 48 49 0C 4A   @A.BC DE FG.HI.J
0x0010: 4B 0D 4C 4D 0A                                    K.LM.
0x0015:
$ 

How to preserve line breaks when storing command output to a variable?

Quote your variables. Here is it why:

$ f="fafafda
> adffd
> adfadf
> adfafd
> afd"

$ echo $f
fafafda adffd adfadf adfafd afd

$ echo "$f"
fafafda
adffd
adfadf
adfafd
afd

Without quotes, the shell replaces $TEMP with the characters it contains (one of which is a newline). Then, before invoking echo shell splits that string into multiple arguments using the Internal Field Separator (IFS), and passes that resulting list of arguments to echo. By default, the IFS is set to whitespace (spaces, tabs, and newlines), so the shell chops your $TEMP string into arguments and it never gets to see the newline, because the shell considers it a separator, just like a space.

How to remove a newline from a string in Bash

Under bash, there are some bashisms:

The tr command could be replaced by // bashism:

COMMAND=$'\nREBOOT\r   \n'
echo "|${COMMAND}|"
|
   OOT
|

echo "|${COMMAND//[$'\t\r\n']}|"
|REBOOT   |

echo "|${COMMAND//[$'\t\r\n ']}|"
|REBOOT|

See Parameter Expansion and QUOTING in bash's man page:

man -Pless\ +/\/pattern bash
man -Pless\ +/\\\'string\\\' bash

man -Pless\ +/^\\\ *Parameter\\\ Exp bash
man -Pless\ +/^\\\ *QUOTING bash

Further...

As asked by @AlexJordan, this will suppress all specified characters. So what if $COMMAND do contain spaces...

COMMAND=$'         \n        RE BOOT      \r           \n'
echo "|$COMMAND|"
|
           BOOT      
|

CLEANED=${COMMAND//[$'\t\r\n']}
echo "|$CLEANED|"
|                 RE BOOT                 |

shopt -q extglob || { echo "Set shell option 'extglob' on.";shopt -s extglob;}

CLEANED=${CLEANED%%*( )}
echo "|$CLEANED|"
|                 RE BOOT|

CLEANED=${CLEANED##*( )}
echo "|$CLEANED|"
|RE BOOT|

Shortly:

COMMAND=$'         \n        RE BOOT      \r           \n'
CLEANED=${COMMAND//[$'\t\r\n']} && CLEANED=${CLEANED%%*( )}
echo "|${CLEANED##*( )}|"
|RE BOOT|

Note: bash have extglob option to be enabled (shopt -s extglob) in order to use *(...) syntax.

shell script how to substitute variable contains newline

Many thaks for relies. Finally, I resolved this by re-assigning flist with flist=`echo $flist`;.

Bash: Strip trailing linebreak from output

If your expected output is a single line, you can simply remove all newline characters from the output. It would not be uncommon to pipe to the tr utility, or to Perl if preferred:

wc -l < log.txt | tr -d '\n'

wc -l < log.txt | perl -pe 'chomp'

You can also use command substitution to remove the trailing newline:

echo -n "$(wc -l < log.txt)"

printf "%s" "$(wc -l < log.txt)"

---

If your expected output may contain multiple lines, you have another decision to make:

If you want to remove MULTIPLE newline characters from the end of the file, again use cmd substitution:

printf "%s" "$(< log.txt)"

If you want to strictly remove THE LAST newline character from a file, use Perl:

perl -pe 'chomp if eof' log.txt

---

Note that if you are certain you have a trailing newline character you want to remove, you can use head from GNU coreutils to select everything except the last byte. This should be quite quick:

head -c -1 log.txt

Also, for completeness, you can quickly check where your newline (or other special) characters are in your file using cat and the 'show-all' flag -A. The dollar sign character will indicate the end of each line:

cat -A log.txt

read line and remove newline character using shell script

You probably generated the text file on a windows machine or some other setting with dos line endings. You can fix that by either

• converting the file to unix line endings with dos2unix
• deleting '\r' characters: cat $FILE | tr -d '\r' | while read LINE ...
• use a utility like awk to grab the first field: cat $FILE | awk '{print $1}' | while read LINE ...

How can I 'echo' out things without a newline?

Yes. Use the -n option:

echo -n "$x"

From help echo:

-n do not append a newline

This would strips off the last newline too, so if you want you can add a final newline after the loop:

for ...; do ...; done; echo

---

Note:

This is not portable among various implementations of echo builtin/external executable. The portable way would be to use printf instead:

printf '%s' "$x"

File content into unix variable with newlines

The assignment does not remove the newline characters, it's actually the echo doing this. You need simply put quotes around the string to maintain those newlines:

echo "$testvar"

This will give the result you want. See the following transcript for a demo:

pax> cat num1.txt ; x=$(cat num1.txt)
line 1
line 2

pax> echo $x ; echo '===' ; echo "$x"
line 1 line 2
===
line 1
line 2

---

The reason why newlines are replaced with spaces is not entirely to do with the echo command, rather it's a combination of things.

When given a command line, bash splits it into words according to the documentation for the IFS variable:

IFS: The Internal Field Separator that is used for word splitting after expansion ... the default value is <space><tab><newline>.

That specifies that, by default, any of those three characters can be used to split your command into individual words. After that, the word separators are gone, all you have left is a list of words.

Combine that with the echo documentation (a bash internal command), and you'll see why the spaces are output:

echo [-neE] [arg ...]: Output the args, separated by spaces, followed by a newline.

When you use echo "$x", it forces the entire x variable to be a single word according to bash, hence it's not split. You can see that with:

pax> function count {
...>    echo $#
...> }
pax> count 1 2 3
3
pax> count a b c d
4
pax> count $x
4
pax> count "$x"
1

Here, the count function simply prints out the number of arguments given. The 1 2 3 and a b c d variants show it in action.

Then we try it with the two variations on the x variable. The one without quotes shows that there are four words, "test", "1", "test" and "2". Adding the quotes makes it one single word "test 1\ntest 2".

How to avoid bash command substitution to remove the newline character?

Non-trailing newlines are not removed

The newlines you are looking for are there, you just don't see them, because you use echo without quoting the variable.

Validation:

$ a=$( df -H )
$ echo $a
Filesystem Size Used Avail Use% Mounted on /dev/sda3 276G 50G 213G 19% / udev 2.1G 4.1k 2.1G 1% /dev tmpfs 832M 820k 832M 1% /run none 5.3M 0 5.3M 0% /run/lock none 2.1G 320k 2.1G 1% /run/shm
$ echo "$a"
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       276G   50G  213G  19% /
udev            2.1G  4.1k  2.1G   1% /dev
tmpfs           832M  820k  832M   1% /run
none            5.3M     0  5.3M   0% /run/lock
none            2.1G  320k  2.1G   1% /run/shm
$ 

Trailing newlines are removed

As @user4815162342 correctly pointed out, although newlines within the output are not removed, trailing newlines are removed with command substitution. See experiment below:

$ a=$'test\n\n'
$ echo "$a"
test


$ b=$(echo "$a")
$ echo "$b"
test
$

In most cases this does not matter, because echo will add the removed newline (unless it is invoked with the -n option), but there are some edge cases where there are more that one trailing newlines in the output of a program, and they are significant for some reason.

Workarounds

1. Add dummy character

In these case, as @Scrutinizer mentioned, you can use the following workaround:

$ a=$(printf 'test\n\n'; printf x); a=${a%x}
$ echo "$a"
test


$ 

Explanation: Character x is added to the output (using printf x), after the newlines. Since the newlines are not trailing any more, they are not removed by the command substitution. The next step is to remove the x we added, using the % operator in ${a%x}. Now we have the original output, with all newlines present!!!

2. Read using process substitution

Instead of using command substitution to assign the output of a program to variable, we can instead use process substitution to feed the output of the program to the read built-in command (credit to @ormaaj). Process substitution preserves all newlines. Reading the output to a variable is a bit tricky, but you can do it like this:

$ IFS= read -rd '' var < <( printf 'test\n\n' ) 
$ echo "$var"
test


$ 

Explanation:

• We set the internal field separator for the read command to null, with IFS=. Otherwise read would not assign the entire output to var, but only the first token.
• We invoke read with options -rd ''. The r is for preventing the backslash to act as a special character, and with d '' set the delimiter to nothing, so that read reads the entire output, instead of just the first line.

3. Read from a pipe

Instead of using command or process substitution to assign the output of a program to variable, we can instead pipe the output of the program to the read command (credit to @ormaaj). Piping also preserves all newlines. Note however, that this time we set the lastpipe shell optional behavior, using the shopt builtin. This is required, so that the read command is executed in the current shell environment. Otherwise, the variable will be assigned in a subshell, and it will not be accessible from the rest of the script.

$ cat test.sh 
#!/bin/bash
shopt -s lastpipe
printf "test\n\n" | IFS= read -rd '' var
echo "$var"
$ ./test.sh 
test


$

---

Related Topics