All Newlines Are Removed When Saving Cat Output into a Variable

All newlines are removed when saving cat output into a variable

The shell is splitting the msgs variable so echo get multiple parameters.
You need to quote your variable to prevent this to happen:

echo "$msgs"

How to preserve line breaks when storing command output to a variable?

Quote your variables. Here is it why:

$ f="fafafda
> adffd
> adfadf
> adfafd
> afd"

$ echo $f
fafafda adffd adfadf adfafd afd

$ echo "$f"
fafafda
adffd
adfadf
adfafd
afd

Without quotes, the shell replaces $TEMP with the characters it contains (one of which is a newline). Then, before invoking echo shell splits that string into multiple arguments using the Internal Field Separator (IFS), and passes that resulting list of arguments to echo. By default, the IFS is set to whitespace (spaces, tabs, and newlines), so the shell chops your $TEMP string into arguments and it never gets to see the newline, because the shell considers it a separator, just like a space.

How to preserve whitespace when saving command output to a Makefile variable?

I want to capture that output into a variable and then later print the
variable to a file.

There does not seem to be a way around make's mechanism to translate newlines in the shell command output into spaces. A bit of a hack that stays close to your original approach would be to have the shell convert newlines into some uncommon character (like \1) when assigning the output to the variable and then have it translate it back when echo-ing that variable to the file. Something like this:

OUTPUT=$(shell cowsay hello | tr '\n' '\1')

all:
        @echo "$(OUTPUT)" | tr '\1' '\n' > output.txt

For me, this results in

$ make
$ cat output.txt 
 _______ 
< hello >
 ------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Why are all newlines gone after PowerShell's Get-Content, Regex, and Set-Content?

For the $replacement variable, you don't really need to specify the type [string], PowerShell will infer that from the assignment.

For the $template variable, [string] is actually wrong. By default, Get-Content will give you an array of strings (i.e. lines) instead of one string.

But in fact you don't even want to split the input into lines in the first place. When Set-Content or Out-File see an array as their input, they will join it with spaces.

Using -Raw makes Get-Content return the entire file as one string, this way also the line endings (like LF for Linux files) will stay the way they are.

$replacement = "Foo Bar"
$template = Get-Content -Path "$pwd\template.sh" -Encoding UTF8 -Raw
$template = $template -replace '<REPLACE_ME>', $replacement
Set-Content -Path "$pwd\script.sh" -Value $template -Encoding UTF8

PowerShell will save all UTF-8 files with a BOM. If you don't want that, you must use a different utility to write the file:

$UTF8_NO_BOM = New-Object System.Text.UTF8Encoding $False

$replacement = "Foo Bar"
$template = Get-Content -Path "$pwd\template.sh" -Encoding UTF8 -Raw
$template = $template -replace '<REPLACE_ME>', $replacement
[System.IO.File]::WriteAllText("$pwd\script.sh", $template, $UTF8_NO_BOM)

Notes:

• PowerShell operators (like -replace) silently operate on arrays. $x -replace "search", "replacement" will perform a replace operation on every member of $x, be that a single string or an array of them.
• Recommended reading: PowerShell Set-Content and Out-File what is the difference?

Bash: redirect `cat` to file without newline

With these inputs:

userInput="Test Test Test"

echo "Line 1
Line 2
Line 3" >file1

echo "Line 4
Line 5
Line 6" >file2

I would do:

printf "%s%s%s" "$(cat file1)" "$userInput" "$(cat file2)" >newfile

The creation of >newfile is equivalent to touch and adding content in your first step. A bit easier to see intent with this.

I get:

$ cat newfile
Line 1
Line 2
Line 3Test Test TestLine 4
Line 5
Line 6

How to avoid bash command substitution to remove the newline character?

Non-trailing newlines are not removed

The newlines you are looking for are there, you just don't see them, because you use echo without quoting the variable.

Validation:

$ a=$( df -H )
$ echo $a
Filesystem Size Used Avail Use% Mounted on /dev/sda3 276G 50G 213G 19% / udev 2.1G 4.1k 2.1G 1% /dev tmpfs 832M 820k 832M 1% /run none 5.3M 0 5.3M 0% /run/lock none 2.1G 320k 2.1G 1% /run/shm
$ echo "$a"
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       276G   50G  213G  19% /
udev            2.1G  4.1k  2.1G   1% /dev
tmpfs           832M  820k  832M   1% /run
none            5.3M     0  5.3M   0% /run/lock
none            2.1G  320k  2.1G   1% /run/shm
$ 

Trailing newlines are removed

As @user4815162342 correctly pointed out, although newlines within the output are not removed, trailing newlines are removed with command substitution. See experiment below:

$ a=$'test\n\n'
$ echo "$a"
test

$ b=$(echo "$a")
$ echo "$b"
test
$

In most cases this does not matter, because echo will add the removed newline (unless it is invoked with the -n option), but there are some edge cases where there are more that one trailing newlines in the output of a program, and they are significant for some reason.

Workarounds

1. Add dummy character

In these case, as @Scrutinizer mentioned, you can use the following workaround:

$ a=$(printf 'test\n\n'; printf x); a=${a%x}
$ echo "$a"
test

$ 

Explanation: Character x is added to the output (using printf x), after the newlines. Since the newlines are not trailing any more, they are not removed by the command substitution. The next step is to remove the x we added, using the % operator in ${a%x}. Now we have the original output, with all newlines present!!!

2. Read using process substitution

Instead of using command substitution to assign the output of a program to variable, we can instead use process substitution to feed the output of the program to the read built-in command (credit to @ormaaj). Process substitution preserves all newlines. Reading the output to a variable is a bit tricky, but you can do it like this:

$ IFS= read -rd '' var < <( printf 'test\n\n' ) 
$ echo "$var"
test

$ 

Explanation:

• We set the internal field separator for the read command to null, with IFS=. Otherwise read would not assign the entire output to var, but only the first token.
• We invoke read with options -rd ''. The r is for preventing the backslash to act as a special character, and with d '' set the delimiter to nothing, so that read reads the entire output, instead of just the first line.

3. Read from a pipe

Instead of using command or process substitution to assign the output of a program to variable, we can instead pipe the output of the program to the read command (credit to @ormaaj). Piping also preserves all newlines. Note however, that this time we set the lastpipe shell optional behavior, using the shopt builtin. This is required, so that the read command is executed in the current shell environment. Otherwise, the variable will be assigned in a subshell, and it will not be accessible from the rest of the script.

$ cat test.sh 
#!/bin/bash
shopt -s lastpipe
printf "test\n\n" | IFS= read -rd '' var
echo "$var"
$ ./test.sh 
test

$

File content into unix variable with newlines

The assignment does not remove the newline characters, it's actually the echo doing this. You need simply put quotes around the string to maintain those newlines:

echo "$testvar"

This will give the result you want. See the following transcript for a demo:

pax> cat num1.txt ; x=$(cat num1.txt)
line 1
line 2

pax> echo $x ; echo '===' ; echo "$x"
line 1 line 2
===
line 1
line 2

---

The reason why newlines are replaced with spaces is not entirely to do with the echo command, rather it's a combination of things.

When given a command line, bash splits it into words according to the documentation for the IFS variable:

IFS: The Internal Field Separator that is used for word splitting after expansion ... the default value is <space><tab><newline>.

That specifies that, by default, any of those three characters can be used to split your command into individual words. After that, the word separators are gone, all you have left is a list of words.

Combine that with the echo documentation (a bash internal command), and you'll see why the spaces are output:

echo [-neE] [arg ...]: Output the args, separated by spaces, followed by a newline.

When you use echo "$x", it forces the entire x variable to be a single word according to bash, hence it's not split. You can see that with:

pax> function count {
...>    echo $#
...> }
pax> count 1 2 3
3
pax> count a b c d
4
pax> count $x
4
pax> count "$x"
1

Here, the count function simply prints out the number of arguments given. The 1 2 3 and a b c d variants show it in action.

Then we try it with the two variations on the x variable. The one without quotes shows that there are four words, "test", "1", "test" and "2". Adding the quotes makes it one single word "test 1\ntest 2".

Capturing multiple line output into a Bash variable

Actually, RESULT contains what you want — to demonstrate:

echo "$RESULT"

What you show is what you get from:

echo $RESULT

---

As noted in the comments, the difference is that (1) the double-quoted version of the variable (echo "$RESULT") preserves internal spacing of the value exactly as it is represented in the variable — newlines, tabs, multiple blanks and all — whereas (2) the unquoted version (echo $RESULT) replaces each sequence of one or more blanks, tabs and newlines with a single space. Thus (1) preserves the shape of the input variable, whereas (2) creates a potentially very long single line of output with 'words' separated by single spaces (where a 'word' is a sequence of non-whitespace characters; there needn't be any alphanumerics in any of the words).

Bash: Strip trailing linebreak from output

If your expected output is a single line, you can simply remove all newline characters from the output. It would not be uncommon to pipe to the tr utility, or to Perl if preferred:

wc -l < log.txt | tr -d '\n'

wc -l < log.txt | perl -pe 'chomp'

You can also use command substitution to remove the trailing newline:

echo -n "$(wc -l < log.txt)"

printf "%s" "$(wc -l < log.txt)"

---

If your expected output may contain multiple lines, you have another decision to make:

If you want to remove MULTIPLE newline characters from the end of the file, again use cmd substitution:

printf "%s" "$(< log.txt)"

If you want to strictly remove THE LAST newline character from a file, use Perl:

perl -pe 'chomp if eof' log.txt

---

Note that if you are certain you have a trailing newline character you want to remove, you can use head from GNU coreutils to select everything except the last byte. This should be quite quick:

head -c -1 log.txt

Also, for completeness, you can quickly check where your newline (or other special) characters are in your file using cat and the 'show-all' flag -A. The dollar sign character will indicate the end of each line:

cat -A log.txt

---

Related Topics