How Does "Cat ≪≪ Eof" Work in Bash

How does cat EOF work in bash?

This is called heredoc format to provide a string into stdin. See https://en.wikipedia.org/wiki/Here_document#Unix_shells for more details.

---

From man bash:

Here Documents

This type of redirection instructs the shell to read input from
the current source until a line
containing only word (with no trailing
blanks) is seen.

All of the lines read up to that point are then used as the
standard input for a command.

The format of here-documents is:

          <<[-]word
                  here-document
          delimiter

No parameter expansion, command substitution, arithmetic expansion, or
pathname expansion is performed on
word. If any characters in word are
quoted, the
delimiter is the result of quote removal on word, and the lines
in the here-document are not expanded.
If word is unquoted, all lines of the
here-document are subjected to parameter expansion, command
substitution, and arithmetic
expansion. In the latter case, the
character sequence \<newline> is
ignored, and \ must be used to quote the characters \, $, and `.

If the redirection operator is <<-, then all leading tab characters
are stripped from input lines and the
line containing delimiter. This
allows here-documents within shell scripts to be indented in a natural fashion.

What does @(cat - EOF do in Bash?

<(...) is a process substitution. Bash creates a fifo in some directory and run the command inside <(...) and substitutes the expression with the fifo name. Process substitutions have (strange) lifetime rules, but they are usually valid till the end of command or line. For example:

$ cmd=<(echo 123); echo cmd=$cmd; cat $cmd
cmd=/dev/fd/63
123

<<-EOF is a here document. If there is - in front of the delimeter then leading tabs on following lines including the line with delimeter are ignored. (Note: stackoverflow doesn't preserve tabs).

$ echo -e '
cat <<EOF
\tblabla
EOF
cat <<-EOF
\t\t\t\t\tblabla
\t\t\t\t\t\t\t\t\t\t\t\t\tEOF
' > file.sh
$ bash ./file.sh
      blabla
blabla

notify @<(...) just substitutes the <(...) part inside for some /dev/fd/<number> and executes notify @/dev/fd/<number>. Probably the @ is used for notify process to indicate it should read from file and the rest of the argument is the filename. Then the cat process that has tied output to /dev/fd/<number> fifo created with process substitution, the cat process receives the here document content { <more json data> } on standard input. cat outputs standard input to output and then I guess notify reads the fifo and receives the characters.

How to cat EOF a file containing code?

You only need a minimal change; single-quote the here-document delimiter after <<.

cat <<'EOF' >> brightup.sh

or equivalently backslash-escape it:

cat <<\EOF >>brightup.sh

Without quoting, the here document will undergo variable substitution, backticks will be evaluated, etc, like you discovered.

If you need to expand some, but not all, values, you need to individually escape the ones you want to prevent.

cat <<EOF >>brightup.sh
#!/bin/sh
# Created on $(date # : <<-- this will be evaluated before cat;)
echo "\$HOME will not be evaluated because it is backslash-escaped"
EOF

will produce

#!/bin/sh
# Created on Fri Feb 16 11:00:18 UTC 2018
echo "$HOME will not be evaluated because it is backslash-escaped"

As suggested by @fedorqui, here is the relevant section from man bash:

Here Documents

This type of redirection instructs the shell to read input from the
current source until a line containing only delimiter (with no
trailing blanks) is seen. All of the lines read up to that point are
then used as the standard input for a command.

The format of here-documents is:

      <<[-]word
              here-document
      delimiter

No parameter expansion, command substitution, arithmetic expansion,
or pathname expansion is performed on word. If any characters in word
are quoted, the delimiter is the result of quote removal on word, and
the lines in the here-document are not expanded. If word is
unquoted, all lines of the here-document are subjected to parameter
expansion, command substitution, and arithmetic expansion. In the
latter case, the character sequence \<newline> is ignored, and \
must be used to quote the characters \, $, and `.

How can I read a file in cat EOF

You could use backticks, or write the file in chunks:

---

#!/bin/bash

cat <<OMG >zfile
ad3
c43
34e
se3
we3
OMG

# Method1 : backticks

cat << EOF > test.sh
#!/bin/bash
some commands1
`cat zfile`
some commands2
EOF

# Method2: append

cat << EOF1 > test2.sh
#!/bin/bash
some commands1
EOF1

cat zfile >> test2.sh

cat << EOF2 >> test2.sh
some commands2
EOF2

Is it safe to use cat -EOF

You should realize that some users set their editors to automatically convert tabs into spaces. So when you work with multiple users on the same script, it is paramount to set up some base rules for this, which is annoying as you will end up with a one-day tab-vs-spaces war. Furthermore, I often consider tabs evil. They are good as a field separator, but they are awful when visualization depends on it. Everything depends on where the tabstops are defined and this can change from editor to editor and terminal to terminal. Sometimes. the tabstops are defined as a multiple of 8, sometimes as a multiple of 4. There is even the Posix command tabs which allows you to set the tabstops in any way you want

$ tabs 1,6,10,14,18,22,26,30,34,38,42,46,50,54,58,62,67

So ... tabs are evil!

My suggestion to you is to use the following:

 printf "%s\n" 'fantastic line1' \
               'awesome line2'   \
               'amazing line3'   \
               'unrivaled line4'

What does this cat command with an operator do?

The technical term for this shell construct is a Here Document ("here" as in "the data is here in this script"). If you had only cat >file.c then stdin would be read until an end-of-file condition (on Unix typically CTRL-D) is met. The result would be written to file.c.

The << EOF part now instructs the shell to pass the following lines on stdin to cat (instead of connecting stdin to the terminal). The shell recognizes the EOF on a line by itself as the end marker. The details are in your shell manual, e.g. for bash it is this paragraph:

Here Documents
   This type of redirection instructs the shell to read input from  the  current  source
   until  a  line  containing  only  word (with no trailing blanks) is seen.  All of the
   lines read up to that point are then used as the standard input for a command.

   The format of here-documents is:

          <<[-]word
                  here-document
          delimiter

   No parameter expansion,  command  substitution,  arithmetic  expansion,  or  pathname
   expansion  is performed on word.  If any characters in word are quoted, the delimiter
   is the result of quote removal on word, and the lines in the  here-document  are  not
   expanded.   If  word  is  unquoted,  all  lines of the here-document are subjected to
   parameter expansion, command substitution, and arithmetic expansion.  In  the  latter
   case,  the  character sequence \<newline> is ignored, and \ must be used to quote the
   characters \, $, and ‘.

   If the redirection operator is <<-, then all leading tab characters are stripped from
   input  lines  and  the  line containing delimiter.  This allows here-documents within
   shell scripts to be indented in a natural fashion.

SSH remote execution - How to declare a variable inside EOF block (Bash script)

The local shell is "expanding" all of the $variable references in the here-document, but AIUI you want $i to be passed through to the remote shell and expanded there. To do this, escape (with a backslash) the $ characters you don't want the local shell to expand. I think it'll look like this:

ssh -i $aws_pem $aws_host << EOF
mysqldump --column-statistics=0 --result-file=$dump_file -u user -p$pwd_prod -h $db_to_bk my_database
mysql -u user -p$pwd_prod -h $db_to_bk -N -e 'SHOW TABLES from my_database' > $remote_home/test.txt
sh -c 'cat test.txt | while read i ; do mysql -u user -p$pwd_prod -h $db_to_bk -D my_database --tee=$remote_home/rows.txt -e "SELECT COUNT(*) as \$i FROM \$i" ; done'
EOF

You can test this by replacing the ssh -i $aws_pem $aws_host command with just cat, so it prints the here-document as it'll be passed to the ssh command (i.e. after the local shell has done its parsing and expansions, but before the remote shell has done its). You should see most of the variables replaced by their values (because those have to happen locally, where those variables are defined) but $i passed literally so the remote shell can expand it.

BTW, you should double-quote almost all of your variable references (e.g. ssh -i "$aws_pem" "$aws_host") to prevent weird parsing problems; shellcheck.net will point this out for the local commands (along with some other potential problems), but you should fix it for the remote commands as well (except $i, since that's already double-quoted as part of the SELECT command).

---

Related Topics