How to Perform a For-Each Loop Over All the Files Under a Specified Path

How to perform a for-each loop over all the files under a specified path?

Here is a better way to loop over files as it handles spaces and newlines in file names:

#!/bin/bash

find . -type f -iname "*.txt" -print0 | while IFS= read -r -d $'\0' line; do
    echo "$line"
    ls -l "$line"    
done

How can I iterate over files in a given directory?

Python 3.6 version of the above answer, using os - assuming that you have the directory path as a str object in a variable called directory_in_str:

import os

directory = os.fsencode(directory_in_str)
    
for file in os.listdir(directory):
     filename = os.fsdecode(file)
     if filename.endswith(".asm") or filename.endswith(".py"): 
         # print(os.path.join(directory, filename))
         continue
     else:
         continue

Or recursively, using pathlib:

from pathlib import Path

pathlist = Path(directory_in_str).glob('**/*.asm')
for path in pathlist:
     # because path is object not string
     path_in_str = str(path)
     # print(path_in_str)

• Use rglob to replace glob('**/*.asm') with rglob('*.asm')
  
  • This is like calling Path.glob() with '**/' added in front of the given relative pattern:

from pathlib import Path

pathlist = Path(directory_in_str).rglob('*.asm')
for path in pathlist:
     # because path is object not string
     path_in_str = str(path)
     # print(path_in_str)

---

Original answer:

import os

for filename in os.listdir("/path/to/dir/"):
    if filename.endswith(".asm") or filename.endswith(".py"): 
         # print(os.path.join(directory, filename))
        continue
    else:
        continue

Iterate all files in a directory using a 'for' loop

This lists all the files (and only the files) in the current directory and its subdirectories recursively:

for /r %i in (*) do echo %i

Also if you run that command in a batch file you need to double the % signs.

for /r %%i in (*) do echo %%i

(thanks @agnul)

How to loop over files in directory and change path and add suffix to filename

A couple of notes first: when you use Data/data1.txt as an argument, should it really be /Data/data1.txt (with a leading slash)? Also, should the outer loop scan only for .txt files, or all files in /Data? Here's an answer, assuming /Data/data1.txt and .txt files only:

#!/bin/bash
for filename in /Data/*.txt; do
    for ((i=0; i<=3; i++)); do
        ./MyProgram.exe "$filename" "Logs/$(basename "$filename" .txt)_Log$i.txt"
    done
done

Notes:

• /Data/*.txt expands to the paths of the text files in /Data (including the /Data/ part)
• $( ... ) runs a shell command and inserts its output at that point in the command line
• basename somepath .txt outputs the base part of somepath, with .txt removed from the end (e.g. /Data/file.txt -> file)

If you needed to run MyProgram with Data/file.txt instead of /Data/file.txt, use "${filename#/}" to remove the leading slash. On the other hand, if it's really Data not /Data you want to scan, just use for filename in Data/*.txt.

Loop through files in directory specified using argument

This would happen if the directory is empty, or misspelled. The shell (in its default configuration) simply doesn't expand a wildcard if it has no matches. (You can control this in Bash with shopt -s nullglob; with this option, wildcards which don't match anything are simply removed.)

You can verify this easily for yourself. In a directory with four files,

sh$ echo *
a       file    or      two

sh$ echo [ot]*
or      two

sh$ echo n*
n*

And in Bash,

bash$ echo n*
n*

bash$ shopt -s nullglob

bash$ echo n*

I'm guessing you are confused about how the current working directory affects the resolution of directory names; maybe read Difference between ./ and ~/

How to loop through all the files located under a certain path in zsh?

There is no need to use find. You could try the following:

for file in /path/to/directory/**/*(.); do echo $file; done

or

for file in /path/to/directory/**/*(.); echo $file

• the ** pattern matches multiple directories recursively. So a/**/b matches any b somewhere below a. It is essentially matches the list find a -name b produces.
• (.) is a glob qualifier and tells zsh to only match plain files. It is the equivalent to the -type f option from find.
• you do not really need double quotes around $file because zsh does not split variables into words on substitution.
• the first version is the regular form of the for-loop; the second one is the short form without do and done

The reason for the error you get is due to the last point: when running a single command in the loop you need either both do and done or none of them. If you want to run more than one command in the loop, you must use them.

For Loop in Python to move through each file sequentially and append it into an array

Your code could simplify (aside from actions, which you don't show us) to something like

import os
from collections import defaultdict

root = "/tmp/mp_data"

# Map labels (subdirectories of root) to data
data_per_label = defaultdict(list)

# Get all top-level directories within `root`
label_dirs = [
    name for name in os.listdir(root) if os.path.isdir(os.path.join(root, name))
]
print(f"{label_dirs=}")

# Loop over each label directory
for label in label_dirs:
    label_dir = os.path.join(root, label)
    # Loop over each filename in the label directory
    for filename in os.listdir(label_dir):
        # Take care to only look at .npy files
        if filename.endswith(".npy"):
            filepath = os.path.join(label_dir, filename)
            print(f"{label=} {filename=} {filepath=}")
            data = filename  # replace with np.load(filename)
            data_per_label[label].append(data)

print(data_per_label)

Given a tree like

/tmp/mp_data
├── r1
│   └── a.npy
├── r2
│   └── b.npy
└── r3
    └── c.npy

this prints out

label_dirs=['r1', 'r3', 'r2']
label='r1' filename='a.npy' filepath='/tmp/mp_data/r1/a.npy'
label='r3' filename='c.npy' filepath='/tmp/mp_data/r3/c.npy'
label='r2' filename='b.npy' filepath='/tmp/mp_data/r2/b.npy'
defaultdict(<class 'list'>, {'r1': ['/tmp/mp_data/r1/a.npy'], 'r3': ['/tmp/mp_data/r3/c.npy'], 'r2': ['/tmp/mp_data/r2/b.npy']})

Loop through all files in all folders recursively as fast as possible in GOLANG

If you don't want to use any external package, you can create a separate worker routine for file processing, then start as many workers you want. After that, go into the tree recursively in your main thread, and send out the jobs to the workers. If any worker "has time", it will pick up the following job from the jobs channel and process it.

var (
    wg   *sync.WaitGroup
    jobs chan string = make(chan string)
)

func loopFilesWorker() error {
    for path := range jobs {
        files, err := ioutil.ReadDir(path)
        if err != nil {
            wg.Done()
            return err
        }

        for _, file := range files {
            if !file.IsDir() {
                fmt.Println(file.Name())
            }
        }
        wg.Done()
    }
    return nil
}

func LoopDirsFiles(path string) error {
    files, err := ioutil.ReadDir(path)
    if err != nil {
        return err
    }
    //Add this path as a job to the workers
    //You must call it in a go routine, since if every worker is busy, then you have to wait for the channel to be free.
    go func() {
        wg.Add(1)
        jobs <- path
    }()
    for _, file := range files {
        if file.IsDir() {
            //Recursively go further in the tree
            LoopDirsFiles(filepath.Join(path, file.Name()))
        }
    }
    return nil
}

func main() {
    //Start as many workers you want, now 10 workers
    for w := 1; w <= 10; w++ {
        go loopFilesWorker()
    }
    //Start the recursion
    LoopDirsFiles(globals.SrcPath)
    wg.Wait()
}

How do I iterate through the files in a directory and it's sub-directories in Java?

You can use File#isDirectory() to test if the given file (path) is a directory. If this is true, then you just call the same method again with its File#listFiles() outcome. This is called recursion.

Here's a basic kickoff example:

package com.stackoverflow.q3154488;

import java.io.File;

public class Demo {

    public static void main(String... args) {
        File dir = new File("/path/to/dir");
        showFiles(dir.listFiles());
    }

    public static void showFiles(File[] files) {
        for (File file : files) {
            if (file.isDirectory()) {
                System.out.println("Directory: " + file.getAbsolutePath());
                showFiles(file.listFiles()); // Calls same method again.
            } else {
                System.out.println("File: " + file.getAbsolutePath());
            }
        }
    }
}

Note that this is sensitive to StackOverflowError when the tree is deeper than the JVM's stack can hold. If you're already on Java 8 or newer, then you'd better use Files#walk() instead which utilizes tail recursion:

package com.stackoverflow.q3154488;

import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

public class DemoWithJava8 {

    public static void main(String... args) throws Exception {
        Path dir = Paths.get("/path/to/dir");
        Files.walk(dir).forEach(path -> showFile(path.toFile()));
    }

    public static void showFile(File file) {
        if (file.isDirectory()) {
            System.out.println("Directory: " + file.getAbsolutePath());
        } else {
            System.out.println("File: " + file.getAbsolutePath());
        }
    }
}

---

Related Topics