Why 'Read' Command in Shell Script Is Missing Initial Characters

why 'read' command in shell script is missing initial characters?

This is pretty well explained in this post I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!. When reading a file line by line, if a command inside the loop also reads stdin, it can exhaust the input file. In your case ffmpeg also reads from stdin.

The most common symptom of this is a while read loop only running once, even though the input contains many lines. This is because the rest of the lines are swallowed by the offending command. The most common fix for the problem is to close the stdin of the ffmpeg by doing < /dev/null

ffmpeg -y -i "$ifile" -c:v libx264rgb -b:v 512k -bf 0 -pix_fmt rgb24  -r 25 -strict -2 "$ofile" < /dev/null

or use another file descriptor other than standard input

 while read -r line <&3; do
     ifile="$line"
     ofile="abc_${line}"
     ffmpeg -y -i "$ifile" -c:v libx264rgb -b:v 512k -bf 0 -pix_fmt rgb24  -r 25 -strict -2 "$ofile"
 done 3<file

Or your problem could altogether be a case of the input file having DOS style line endings carried over from a DOS environment. You can check that out by running the file command on the input file (file file_list) which could show CRLF line terminators. In such case do a clean-up of the input file as dos2unix file_list and re-run your script.

Bash script removing first letter

The reason you're missing the first character of the usernames is precisely the same as the reason your pause doesn't work: your pause (read -n 1 -p) reads one character from stdin. However, stdin has been redirected to the datafile, so it reads the first character of the username.

You really need to quote variables in expansions. Get in the habit of doing so. It's much safer.

I don't know what the replacement line does either: as you present it, it looks like a no-op but I suppose it's possible that one of the spaces is a tab. Anyway, it is unnecessary. You can replace all of that fooling around with shell variables like this:

while read -r VARUSER VARPASS ignore_rest_of_line; do
  echo "Username: $VARUSER Password: $VARPASS"
  imapsync --host1 123.456.789.123 --user1 "$VARUSER" \
           --password1 "$VARPASS" --authmech1 PLAIN \
           --host2 999.888.777.123 \
           --user2 "$VARUSER*masteruser@mydomain.com" \
           --password2 notreallypasswordXD \
           --authmech2 PLAIN --exclude 'Trash|Spam'
  echo "Done!"
  echo
  read -n 1 -s < /dev/tty
done < mailusers.cfg

first character of string in shell not working

It works if while is replaced with an if-then statement:

while IFS=$'\t' read -r -a myArray
do 
    like=${myArray[0]}
    position=${myArray[1]}

    if [ "${like:0:1}" == "E" ]
    then
        file=$like."Rput"
        echo "$file"
    fi
done

Note also that spaces are important. The following tests first character of like for equality with E:

[ "${like:0:1}" == "E" ]

But, the following does something unrelated:

[ "${like:0:1}"=="E" ]

Since the equal sign here is not separated by spaces, "${like:0:1}"=="E" is interpreted simply as a single string. Tests of a single string return true if the string is nonempty and false if it is empty. Since ${like:0:1}"=="E" is always non-empty, it will always return true.

Bash: Special characters lost when reading file

You lost the backslashes and spaces because bash (via its read builtin) is evaluating the value of the text - substituting variables, looking for escape characters (tab, newline), etc. See the manpage for some details. Also, echo will combine whitespace.

As far as preserving them, I'm not sure you can. You'd probably get the backslashes back by doing:

while read -r line
  do
    echo $line
  done < "my.tex"

which should modify read to not try to evaluate the backslashes. It will probably still swallow the leading spaces, though.

Edit: setting the $IFS special variable to the empty string, like this:

export IFS=

will cause the spaces to be preserved in this case.

character missing when reading from file in shell script

while IFS= read -r line; do
    echo "$line"
done < namespaces.txt

This is the correct way to read and print a line at a time. "$line" is quoted, and IFS can be set to empty.

Check for IFS=\n, or similar, in your script. Somewhere, you have set IFS to contain n. Then when you print $line with no quotes, it gets split in to multiple words by IFS (n), and echo prints each word given to it, with a space in between.

While loop stops reading after the first line in Bash

The problem is that do_work.sh runs ssh commands and by default ssh reads from stdin which is your input file. As a result, you only see the first line processed, because the command consumes the rest of the file and your while loop terminates.

This happens not just for ssh, but for any command that reads stdin, including mplayer, ffmpeg, HandBrakeCLI, httpie, brew install, and more.

To prevent this, pass the -n option to your ssh command to make it read from /dev/null instead of stdin. Other commands have similar flags, or you can universally use < /dev/null.

Bash read ignores leading spaces

This is covered in the Bash FAQ entry on reading data line-by-line.

The read command modifies each line read; by default it removes all leading and trailing whitespace characters (spaces and tabs, or any whitespace characters present in IFS). If that is not desired, the IFS variable has to be cleared:

# Exact lines, no trimming
while IFS= read -r line; do
  printf '%s\n' "$line"
done < "$file"

As Charles Duffy correctly points out (and I'd missed by focusing on the IFS issue); if you want to see the spaces in your output you also need to quote the variable when you use it or the shell will, once again, drop the whitespace.

Notes about some of the other differences in that quoted snippet as compared to your original code.

The use of the -r argument to read is covered in a single sentence at the top of the previously linked page.

The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines). Without this option, any backslashes in the input will be discarded. You should almost always use the -r option with read.

As to using printf instead of echo there the behavior of echo is, somewhat unfortunately, not portably consistent across all environments and the differences can be awkward to deal with. printf on the other hand is consistent and can be used entirely robustly.

Bash script prints Command Not Found on empty lines

Make sure your first line is:

#!/bin/bash

Enter your path to bash if it is not /bin/bash

---

Try running:

dos2unix script.sh

That wil convert line endings, etc from Windows to unix format. i.e. it strips \r (CR) from line endings to change them from \r\n (CR+LF) to \n (LF).

More details about the dos2unix command (man page)

---

Another way to tell if your file is in dos/Win format:

cat scriptname.sh | sed 's/\r/<CR>/'

The output will look something like this:

#!/bin/sh<CR>
<CR>
echo Hello World<CR>
<CR>

This will output the entire file text with <CR> displayed for each \r character in the file.

---

Related Topics