Why Does Cat <<< $Var1 Lose Newlines

Assign string containing null-character (\0) to a variable in Bash

In Bash, you can't store the NULL-character in a variable.

You may, however, store a plain hex dump of the data (and later reverse this operation again) by using the xxd command.

VAR1=`echo -ne "n\0m\0k" | xxd -p | tr -d '\n'`
echo -ne "$VAR1" | xxd -r -p | od -c   # -> 0000000    n  \0   m  \0   k

How to remove last n characters from a string in Bash?

First, it's usually better to be explicit about your intent. So if you know the string ends in a .rtf that you want to remove, you can just use var2=${var%.rtf}. One potentially-useful aspect of this approach is that if the string doesn't end in .rtf, it is not changed at all; var2 will contain an unmodified copy of var.

If you want to remove a filename suffix but don't know or care exactly what it is, you can use var2=${var%.*} to remove everything starting with the last .. Or, if you only want to keep everything up to but not including the first ., you can use var2=${var%%.*}. Those options have the same result if there's only one . in the string, but if there might be more than one, you get to pick which end of the string to work from. On the other hand, if there's no . in the string at all, var2 will again be an unchanged copy of var.

If you really want to always remove a specific number of characters, here are some options.

You tagged this bash specifically, so we'll start with bash builtins. The one which has worked the longest is the same suffix-removal syntax I used above: to remove four characters, use var2=${var%????}. Or to remove four characters only if the first one is a dot, use var2=${var%.???}, which is like var2=${var%.*} but only removes the suffix if the part after the dot is exactly three characters. As you can see, to count characters this way, you need one question mark per unknown character removed, so this approach gets unwieldy for larger substring lengths.

An option in newer shell versions is substring extraction: var2=${var:0:${#var}-4}. Here you can put any number in place of the 4 to remove a different number of characters. The ${#var} is replaced by the length of the string, so this is actually asking to extract and keep (length - 4) characters starting with the first one (at index 0). With this approach, you lose the option to make the change only if the string matches a pattern. As long as the string has at least four characters, no matter what its actual value is, the copy will include all but its last four characters.

You can leave the start index out; it defaults to 0, so you can shorten that to just var2=${var::${#var}-4}. In fact, newer versions of bash (specifically 4+, which means the one that ships with MacOS won't work) recognize negative lengths as the index of the character to stop at, counting back from the end of the string. So in those versions you can get rid of the string-length expression, too: var2=${var::-4}. This interpretation is also triggered if you leave the string length in but the string is shorter than four characters, since then ${#var}-4 is negative. For example, if the string has three characters, ${var:0:${#var}-4} becomes ${var:0:-1} and removes only the last character.

If you're not actually using bash but some other POSIX-type shell, the pattern-based suffix removal with % will still work – even in plain old dash, where the index-based substring extraction won't. Ksh and zsh do both support substring extraction, but require the explicit 0 start index; zsh also supports the negative end index, while ksh requires the length expression. Note that zsh, which indexes arrays starting at 1, nonetheless indexes strings starting at 0 if you use this bash-compatible syntax. But zsh also allows you to treat scalar parameters as if they were arrays of characters, in which case the substring syntax uses a 1-based count and places the start and (inclusive) end positions in brackets separated by commas: var2=$var[1,-5].

Instead of using built-in shell parameter expansion, you can of course run some utility program to modify the string and capture its output with command substitution. There are several commands that will work; one is var2=$(sed 's/.\{4\}$//' <<<"$var").

Why a variable assignment replaces tabs with spaces

You need to quote your variable $res for whitespace to be preserved.

$ cat file
a       b e     c       d

$ res=$(cat file)

$ echo $res
a b e c d

$ echo "$res"
a       b e     c       d

From man bash under QUOTING:

Quoting is used to remove the special meaning of certain characters
or words to the shell.
Quoting can be used to disable special treatment for special characters, to prevent
reserved words from being recognized as such, and to prevent parameter expansion.

Each of the metacharacters listed above under DEFINITIONS has special meaning to the shell
and must be quoted if it is to represent itself.

...

\a     alert (bell)
\b     backspace
\e
\E     an escape character
\f     form feed
\n     new line
\r     carriage return
\t     horizontal tab
\v     vertical tab
\\     backslash
\'     single quote
\"     double quote
\nnn   the eight-bit character whose value is the octal value nnn
\xHH   the eight-bit character whose value is the hexadecimal value HH
\cx    a control-x character

...

Shell scripting input redirection oddities

A recent addition to bash is the lastpipe option, which allows the last command in a pipeline to run in the current shell, not a subshell, when job control is deactivated.

#!/bin/bash
set +m      # Deactiveate job control
shopt -s lastpipe
echo "hello world" | read var1 var2
echo $var1
echo $var2

will indeed output

hello
world

How to format bash SED/AWK/Per output for further processing

The data happens to have valid tcl list syntax:

set f [open "input.file"]
set data [dict create {*}[read $f]]
close $f

set name [string trim [dict get $data product name]]
dict for {key val} [dict get $data product customers] {
    lappend customers [format "%s{%s}" $key [string trim $val]]
}

set f [open "output.csv" w]
puts $f "product,customers,another_column"
puts $f [join [list $name [join $customers] "something_else"] ,]
close $f

creates output.csv with

product,customers,another_column
thing1,mary{} freddy{} bob{spouse betty},something_else

---

Related Topics