Iterating Over Each Line of Ls -L Output

Iterating over each line of ls -l output

Set IFS to newline, like this:

IFS='
'
for x in `ls -l $1`; do echo $x; done

Put a sub-shell around it if you don't want to set IFS permanently:

(IFS='
'
for x in `ls -l $1`; do echo $x; done)

Or use while | read instead:

ls -l $1 | while read x; do echo $x; done

One more option, which runs the while/read at the same shell level:

while read x; do echo $x; done << EOF
$(ls -l $1)
EOF

Going through each ls line

Using your method, you can calculate the totals using uniq, for example:

$ ls -l | cut -c1-1 | sort | uniq -c
    214 -
     13 d
      2 l
      1 t

uniq -c counts the number of consecutive occurrences of a line, and sort just puts them into some sorted order so that the same types end up together.

If you want these results in variables, then something this would be easier:

dirs=0
files=0

for name in *
do
    if [[ -d "$name" ]]
    then
        ((dirs++))
    elif [[ -f "$name" ]]
    then
        ((files++))
    # Possibly other things you want to count ...
    fi
done

echo "Files: $files"
echo "Directories: $dirs"

How to parse the output of `ls -l` into multiple variables in bash?

for loop splits when it sees any whitespace like space, tab, or newline. So, IFS is needed before loop, (there are a lot of questions about ...)

IFS=$'\n' && for i in $(ncftpls -l 'ftp://theftpserver/path/to/files' | awk '{print $9, $5}'); do

echo $i | awk '{print $NF}' # filesize 
echo $i | awk '{NF--; print}' # filename
# you may have spaces in filenames, so is better to use last column for awk

done

The better way I think is to use while not for, so

ls -l | while read i
do
echo $i | awk '{print $9, $5}'

#split them if you want 
x=echo $i | awk '{print $5}'
y=echo $i | awk '{print $9}'

done

Iterate over lines instead of words in a for loop of shell script

The for loop is not designed to loop over "lines". Instead it loops over "words".

Short terminology: "lines" are things separated by newlines. "words" are things separated by spaces (and newlines, among others). in bash lingo "words" are called "fields".

The idiomatic way to loop over lines is to use a while loop in combination with read.

ioscan -m dsf | while read -r line
do
  printf '%s\n' "$line"
done

Note that the while loop is in a subshell because of the pipe. This can cause some confusion with variable scope. In bash you can work around this by using process substitution.

while read -r line
do
  printf '%s\n' "$line"
done < <(ioscan -m dsf)

see also http://mywiki.wooledge.org/BashFAQ/024

The for loop splits the things to loop over using the characters in the $IFS variable as separators. IFS is short for Internal Field Separator. Usually $IFS contains a space, a tab, and a newline. That means the for loop will loop over the "words", not over the lines.

If you insist on using a for loop to loop over lines you have to change the value of $IFS to only newline. But if you do this you have to save the old value of $IFS and restore that after the loop, because many other things also depend on $IFS.

OLDIFS="$IFS"
IFS=$'\n' # bash specific
for line in $(ioscan -m dsf)
do
  printf '%s\n' "$line"
done
IFS="$OLDIFS"

in POSIX shells, that have no ANSI-C Quoting ($'\n'), you can do it like this:

IFS='
'

that is: put an actual new line between the quotes.

Alternatively you can use a subshell to contain the change to $IFS:

(
  # changes to variables in the subshell stay in the subshell
  IFS=$'\n'
  for line in $(ioscan -m dsf)
  do
    printf '%s\n' "$line"
  done
)
# $IFS is not changed outside of the subshell

But beware the command in the loop may itself depends on some sane setting for $IFS. Then you have to restore the $IFS before executing the command and set again before the next loop or some such. I do not recommend messing with $IFS. Too many commands depend on some sane values in $IFS and changing it is an endless nightmare of obscure bug hunting.

How to apply shell command to each line of a command output?

It's probably easiest to use xargs. In your case:

ls -1 | xargs -L1 echo

The -L flag ensures the input is read properly. From the man page of xargs:

-L number
    Call utility for every number non-empty lines read. 
    A line ending with a space continues to the next non-empty line. [...]

Iterating over very long lines in a huge file

Cutting content down to the first 250 characters of each line is best done before getting into the shell loop. You can do that, while still retaining state from variables set within the loop, via process substitution:

#!/usr/bin/env bash

while IFS= read -r p; do
  : put your code for processing "$p" here
done < <(cut -c -250 ./log.txt)

See BashFAQ #1 discussing the reasons for IFS= and read -r, and BashFAQ #24 for more details on why process substitution is beneficial here.

parse result of `ls -l` with bash script

A better way would be to use bash globbing

Just listing all files

echo *

Or doing something with them

for file in *; do
  echo "$file" # or do something else
done

Or recursively with bash 4+

shopt -s globstar
for file in **/*; do
  echo "$file" # or do something else
done

Update to get directory name and append it to all files within it

Replace mv with an echo to test what it does. Also note ${file##*.} assumes the extension is everything after the last period, so if you had a file like file.tar.gz in directory on, below would turn it into file.tar_on.gz. As far as I know there is no easy way to handle this problem, though you could skip files with multiple . if you want)

#!/bin/bash
d="/some/dir/to/do/this/on"
name=${d##*/} #name=on
for file in "$d"/*; do
  extension=${file##*.} 
  filename=${file%.*}
  filename=${filename##*/}
  [[ -f $file ]] && mv "$file" "$d/${filename}_${name}.$extension"
done

e.g.

> ls /some/dir/to/do/this/on
video1.mpeg  Video2.wmv
> ./abovescript
> ls /some/dir/to/do/this/on
video1_on.mpeg  Video2_on.wmv

Explanation

In bash you can do this

${parameter#word} Removes shortest matching prefix
${parameter##word} Removes longest matching prefix
${parameter%word} Removes shortest matching suffix
${parameter%%word} Removes longest matching suffix

To remove everything anything (*) before and including the last period, I did below

 extension=${file##*.}

To remove everything including and from the last period, I did below (think about shortest match here as going from right to left, e.g. * looks for any non-period text right to left, then when it finds a period it removes that whole section)

filename=${file%.*}

To remove everything up to and including the last /, I did below.

filename=${filename##*/}

Some other notes:

"$d/${filename}_${name}.$extension" Variables can have _ so I switched syntax for a couple of variables here for it to work
"$d"/* Expands to every file of any type (regular, dir, symlink etc...) directly in "$d"

Iterating Over Each Line of Ls -L Output