How to Load All Modules in a Folder

How to load all modules in a folder?

List all python (.py) files in the current folder and put them as __all__ variable in __init__.py

from os.path import dirname, basename, isfile, join
import glob
modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')]

Python: import every module from a folder?

If transforming the folder itself in a module, through the use of a __init__.py file and using from <foldername> import * suits you, you can iterate over the folder contents
with "os.listdir" or "glob.glob", and import each file ending in ".py" with the __import__ built-in function:

import os
for name in os.listdir("plugins"):
    if name.endswith(".py"):
          #strip the extension
         module = name[:-3]
         # set the module name in the current global name space:
         globals()[module] = __import__(os.path.join("plugins", name)

The benefit of this approach is: it allows you to dynamically pass the module names to __import__ - while the ìmport statement needs the module names to be hardcoded, and it allows you to check other things about the files - maybe size, or if they import certain required modules, before importing them.

How do I load all modules under a subdirectory in Python?

This confusion happened, in part, because your module names are the same as the names of the classes you want to load from them. (At least, that's what makes it more confusing.) Your code does correctly load the modules that your classes are in. However, it doesn't load the classes out of those modules, and this is what you actually wanted to do.

Because your class DominatingSets is in the module lib.DominatingSets, its full path from root is lib.DominatingSets.DominatingSets.

from lib import *

in your case will do the same thing as

from lib import DominatingSets
from lib import AnalyzeGraph
# ...

However,

from lib import DominatingSets

is equivalent to

import lib.DominatingSets
DominatingSets = lib.DominatingSets

but lib.DominatingSets is a module (lib/DominatingSets.py), not the class you want.

from lib.DominatingSets import DominatingSets

is equivalent to

import lib.DominatingSets
DominatingSets = lib.DominatingSets.DominatingSets

which is why it works: this is the class you want imported into the name DominatingSets.

If you want to have from lib import * import all the classes in the submodules, you need to import these classes into the lib module. For example, in lib/__init__.py:

from DominatingSets import *
from AnalyzeGraph import *
# ...

While you're making changes, I'd suggest (as others have) using normal Python naming conventions, and have your module names in lowercase: change DominatingSets.py to dominatingsets.py. Then this code would become

from dominatingsets import *
from analyzegraph import *
# ...

how to import all files from different folder in Python

If you have the following structure:

$ tree subdirectory/
subdirectory/
├── file1.py
├── file2.py
└── file3.py

and you want a program to automatically pick up every module which is located in this subdirectory and process it in a certain way you could achieve it as follows:

import glob

# Get file paths of all modules.
modules = glob.glob('subdirectory/*.py')

# Dynamically load those modules here.

For how to dynamically load a module see this question.

---

In your subdirectory/__init__.py you can import all local modules via:

from . import file1
from . import file2
# And so on.

You can import the content of local modules via

from .file1 import *
# And so on.

Then you can import those modules (or the contents) via

from subdirectory import *

With the attribute __all__ in __init__.py you can control what exactly will be imported during a from ... import * statement. So if you don't want file2.py to be imported for example you could do:

__all__ = ['file1', 'file3', ...]

You can access those modules via

import subdirectory
from subdirectory import *

for name in subdirectory.__all__:
    module = locals()[name]

Import and run all modules in folder

You can use importlib. Assume the following simple directory structure:

• a.py
• b.py
• c.py

a.py and b.py contain the following simple function:

def main(name):
    print name

In c.py we can iterate over our directory and use importlib.import_module to import each file. We must ensure to make the imported modules globally accessible, otherwise they will only be local to the for loop.

c.py:

import importlib

files = ['a', 'b']

for f in files:
    globals()[f] = importlib.import_module(f)

a.main('adam')
b.main('ben') 

Running c.py produces the following output:

adam
ben

How do I dynamically import all *.py files from a given directory AND all sub-directories?

Your problem is broken up into two steps (from my point of view):

1. Walk through your directories and sub-directories and find the names of all the .py files you want to import

2. Import them!

To solve (1), we can use the os.walk method to find all the .py files. This function would look like this:

import os
def get_py_files(src):
    cwd = os.getcwd() # Current Working directory
    py_files = [] 
    for root, dirs, files in os.walk(src):
        for file in files:
            if file.endswith(".py"):
                py_files.append(os.path.join(cwd, root, file))
    return py_files

Now that you have a list of your .py files, we need a function that can dynamically import them.

import importlib
def dynamic_import(module_name, py_path):
    module_spec = importlib.util.spec_from_file_location(module_name, py_path)
    module = importlib.util.module_from_spec(module_spec)
    module_spec.loader.exec_module(module)
    return module

And now we just need to put it all together, writing a function that calls your get_py_files function and then loops over the results, loading the modules with dynamic_import.

I am assuming you want the module names that you use in your python script to be the same as the file name, however you can change that by modifying the module_name variable in the function below.

def dynamic_import_from_src(src, star_import = False):
    my_py_files = get_py_files(src)
    for py_file in my_py_files:
        module_name = os.path.split(py_file)[-1].strip(".py")
        imported_module = dynamic_import(module_name, py_file)
        if star_import:
            for obj in dir(imported_module):
                globals()[obj] = imported_module.__dict__[obj]
        else:
            globals()[module_name] = imported_module
    return

Notice we have to call globals() to add the module to the global namespace. Without doing this, the module will be imported but you will have no way of accessing anything inside it. You can pass star_import = True to dynamic_import_from_src if you want it to be a star import instead. (like from first.foo import *. Note that this may overwrite variables in your namespace but that's one of the cons of using the * import anyways.

Throwing it all in one block so it's easier to see it all at once:

import os
import importlib

def get_py_files(src):
    cwd = os.getcwd() # Current Working directory
    py_files = [] 
    for root, dirs, files in os.walk(src):
        for file in files:
            if file.endswith(".py"):
                py_files.append(os.path.join(cwd, root, file))
    return py_files

def dynamic_import(module_name, py_path):
    module_spec = importlib.util.spec_from_file_location(module_name, py_path)
    module = importlib.util.module_from_spec(module_spec)
    module_spec.loader.exec_module(module)
    return module

def dynamic_import_from_src(src, star_import = False):
    my_py_files = get_py_files(src)
    for py_file in my_py_files:
        module_name = os.path.split(py_file)[-1].strip(".py")
        imported_module = dynamic_import(module_name, py_file)
        if star_import:
            for obj in dir(imported_module):
                globals()[obj] = imported_module.__dict__[obj]
        else:
            globals()[module_name] = imported_module
    return

if __name__ == "__main__":
    dynamic_import_from_src("first", star_import = False)

At this point, you can access any of the modules you imported the exact same way as if you did import first.whatever. For example, if first/numbers/one.py contained x=1, then you would be able to access that by saying one.x.

Importing modules from parent folder

It seems that the problem is not related to the module being in a parent directory or anything like that.

You need to add the directory that contains ptdraft to PYTHONPATH

You said that import nib worked with you, that probably means that you added ptdraft itself (not its parent) to PYTHONPATH.

Importing dynamically all modules from a folder

Python does not automatically import submodules contained in a package.
Hence import tracks only loads tracks/__init__.py.

However you can put code inside the __init__.py file that imports all the modules it finds in that directory.

For example putting something like this in the __init__.py:

import os
import importlib

__globals = globals()

for file in os.listdir(os.path.dirname(__file__)):
    mod_name = file[:-3]   # strip .py at the end
    __globals[mod_name] = importlib.import_module('.' + mod_name, package=__name__)

Should make your submodules available as tracks.trackX when importing only tracks.

Or you could use exec:

import os
import importlib

for file in os.listdir(os.path.dirname(__file__)):
    mod_name = file[:-3]   # strip .py at the end
    exec('import .' + mod_name)

---

A cleaner approach would be to use import hooks or implement your own custom module importer. There are multiple ways to do this using importlib see also sys.path_hooks

---

Related Topics