Interacting with Files from Multiple Directories via Bash Script

Interacting with files from multiple directories via bash script

You already accepted the answer of @Charles Duffy but (if I understood well) your question is about having files in different directories so if you need to work with multiple csv files on multiple directories you can use the following snippet:

# array containing the different directories to work with
pathDir=("/foo/bar/dir1" "/buzz/fizz/dir2")

for dir in "${pathDir[@]}" # For each directory
do
    for file in "$dir/"*.csv; do # For each csv file of the directory

        if [[ $(file -i "$file") == "$file: $charset" ]]; then
            iconv -f ISO-8859-1 -t UTF-8 "$file" > "$file.new";
            mv -f "$file.new" "$file";
        fi

    done
done

The pathDir variable is an array which contains the path of different directories.

The first for loop iterate through this array to get all the paths to check.

The second for loop as in the previous answer iterate through the files of the current tested directory.

Using bash to loop through nested folders to run script in current working directory

Another, perhaps more flexible, approach to the problem is to use the find command with the -exec option to run a short "helper-script" for each file found below a directory path that ends in ".7". The -name option allows find to locate all files ending in ".7" below a given directory using simple file-globbing (wildcards). The helper-script then performs the same operation on each file found by find and handles moving the result.h5 to the proper directory.

The form of the command will be:

find /path/to/search -type f -name "*.7" -exec /path/to/helper-script '{}` \;

Where the -f option tells find to only return files (not directories) ending in ".7". Your helper-script needs to be executable (e.g. chmod +x helper-script) and unless it is in your PATH, you must provide the full path to the script in the find command. The '{}' will be replaced by the filename (including relative path) and passed as an argument to your helper-script. The \; simply terminates the command executed by -exec.

(note there is another form for -exec called -execdir and another terminator '+' that can be used to process the command on all files in a given directory -- that is a bit safer, but has additional PATH requirements for the command being run. Since you have only one ".7" file per-directory -- there isn't much benefit here)

The helper-script just does what you need to do in each directory. Based on your description it could be something like the following:

#!/bin/bash

dir="${1%/*}"     ## trim file.7 from end of path
cd "$dir" || {    ## change to directory or handle error
  printf "unable to change to directory %s\n" "$dir" >&2
  exit 1
}

destdir="/Result_Folder/$dir"   ## set destination dir for result.h5
mkdir -p "$destdir" || {        ## create with all parent dirs or exit
  printf "unable to create directory %s\n" "$dir" >&2
  exit 1
}

ls *.pc 2>/dev/null || exit 1   ## check .pc file exists or exit

file7="${1##*/}"  ## trim path from file.7 name

pc_script -f "$file7" -flags1 -other_flags    ## first run

## check result.h5 exists and non-empty and copy to destdir
[ -s "result.h5" ] && cp -a "result.h5" "$destdir/new_result1.h5"

pc_script -f "$file7" -flags2 -other_flags    ## second run

## check result.h5 exists and non-empty and copy to destdir
[ -s "result.h5" ] && cp -a "result.h5" "$destdir/new_result2.h5"

Which essentially stores the path part of the file.7 argument in dir and changes to that directory. If unable to change to the directory (due to read-permissions, etc..) the error is handled and the script exits. Next the full directory structure is created below your Result_Folder with mkdir -p with the same error handling if the directory cannot be created.

ls is used as a simple check to verify that a file ending in ".pc" exits in that directory. There are other ways to do this by piping the results to wc -l, but that spawns additional subshells that are best avoided.

(also note that Linux and Mac have files ending in ".pc" for use by pkg-config used when building programs from source -- they should not conflict with your files -- but be aware they exists in case you start chasing why weird ".pc" files are found)

After all tests are performed, the path is trimmed from the current ".7" filename storing just the filename in file7. The file7 variabli is then used in your pc_script command (which should also include the full path to the script if not in you PATH). After the pc_script is run [ -s "result.h5" ] is used to verify that result.h5 exists and is non-empty before moving that file to your Result_Folder location.

That should get you started. Using find to locate all .7 files is a simple way to let the tool designed to find the files for you do its job -- rather than trying to hand-roll your own solution. That way you only have to concentrate on what should be done for each file found. (note: I don't have pc_script or the files, so I have not testes this end-to-end, but it should be very close if not right-on-the-money)

There is nothing wrong in writing your own routine, but using find eliminates a lot of area where bugs can hide in your own solution.

Let me know if you have further questions.

BASH Script for creating multiple directories, moving files, and then renaming said files

An alternative would be to create symlinks to the original files.

As you said before, each ZMAT symlink would need to be in its own directory.

The upside is that the original data doesn't move, so less risk of breaking it, but the tool you want to use should read the symlinks as if they are the files it is looking for.

This one-liner creates an out directory in the current folder that you could subsequently move wherever you want it. You could easily create it where you do want it by replacing "out" with whatever absolute path you wanted

for i in *.ZMAT; do mkdir -p out/$i ; ln -s $PWD/$i out/$i/ZMAT ; done

Script working on all files in various subdirectories

One solution would use find and xargs:

find /home/working_dir_with_subdirs/ -type f -print0 |
  xargs -0 -n1 /opt/my_script -e -x

This would call /opt/my_script once for each file located in /home/working_dir_with_subdirs and its subdirectories.

If your script can accept multiple files as arguments, you could drop the -n1:

find /home/working_dir_with_subdirs/ -type f -print0 |
  xargs -0 /opt/my_script -e -x

This would call your script with multiple files as arguments, rather than once per file.

In both cases, the -print0 on find and the -0 on xargs are to correctly handle filenames that contain whitespace.

How to move all the files of a directory into multiple directories with a given number of files?

The following should work:

dest_base="destination"
src_dir="src/"

filesperdir=500
atfile=0
atdir=0
for file in $src_dir/*; do
    if ((atfile == 0)); then
        dest_dir=$(printf "$dest_base/%0.5d" $atdir)
        [[ -d $dest_dir ]] || mkdir -p $dest_dir
    fi
    mv $file $dest_dir
    ((atfile++))
    if ((atfile >= filesperdir)); then
        atfile=0
        ((atdir++))
    fi
done

cd to multiple directories in a shell script

You cannot cd to more than one directory at a time (so you need to come back to the original directory, if the path to cd is relative). It is not a limitation of the shell but a property of unix processes (so coding in some other language like Python or C won't change much); they each have only
one current directory (and the shell runs the chdir(2) system call for its cd builtin)

You might consider computing the full path (using realpath(1)) of every directory, and storing that path in some shell variable. Then you could cd in a loop.

Some shells have a stack of directories, which you could manipulate with pushd & popd. In that case, you might code

for d in */* ; do
  pushd "$d"
  sh small_script
  popd
done

You need to be sure that the globbing expansion of */* gives only directories. With zsh you might code */*(/) to expand only directories. See glob(7) & path_resolution(7) & credentials(7).

If your shell don't have pushd you could try

for d in $(realpath */*); do
  cd "$d" && sh small_script.sh
done

or using a subshell

for d in */*; do
  (cd "$d" && exec small_script.sh)
done

See this for more about pushd & popd

Notice that if you do cd a/b followed later by cd e/f the shell is trying to go (for the second cd) into the a/b/e/f directory (which probably do not exist) because e/f is a relative path (so interpreted in the context of the previous current directory a/b). You might want cd a/b followed by cd ../../e/f ...

Take time to understand why cd needs to be a shell builtin. Perhaps read Advanced Linux Programming.

See also find(1). It might be more useful (with its -exec action).

BTW, it could be simpler to change your small_script to accept as argument the directory in which it should go. Then that script might start with cd "$1".

Edit certain files in multiple directories using a shell script

Can you use find to match the user files ?
e.g.

find ./ -maxdepth 1 -name "user*" -exec script_that_does_text_replace.sh {\} \;

where script_that_does_text_replace.sh is the code you wrote above.

Perform an action in every sub-directory using Bash

for D in `find . -type d`
do
    //Do whatever you need with D
done

list all files in multiple directories

You can list file with full path of a given directory using printf:

printf "$PWD%s\n" user.newskims.131017222/*

---

Related Topics