How to Install and Import Python Modules at Runtime

How to install and import Python modules at runtime?

I solved my problem using the imp module.

#!/usr/bin/env python

import pip
import imp

def install_and_load(package):
    pip.main(['install', package])

    path = '/usr/local/lib/python2.7/dist-packages'
    if path not in sys.path:
        sys.path.append(path)

    f, fname, desc = imp.find_module(package)
    return imp.load(package, f, fname, desc)

if __name__ == "__main__":
    try:
        import pexpect
    except:
        pexpect = install_and_load('pexpect')

    # More code...

Actually the code is less than ideal, since I need to hardcode the Python module directory. But since the script is intended for a known target system, I think that is ok.

Import runtime installed module using pip in python 3

By default, at startup Python adds the user site-packages dir (I'm going to refer to it as USPD) in the module search paths. But this only happens if the directory exists on the file system (disk). I didn't find any official documentation to support this statement ¹, so I spent some time debugging and wondering why things seem to be so weird.

The above behavior has a major impact on this particular scenario (pip install --user). Considering the state (at startup) of the Python process that will install modules:

1. USPD exists:

   • Things are straightforward, everything works OK

2. USPD doesn't exist:

   • Module installation will create it
   • But, since it's not in the module search paths, all the modules installed there won't be available for (simple) import statements

When another Python process is started, it will fall under #1.

To fix things, USPD should be manually added to module search paths. Here's how the (beginning of the) script should look like:

import sys
import os
import subprocess
import site

user_site = site.getusersitepackages()
if user_site not in sys.path:
    sys.path.append(user_site)

# ...

@EDIT0:

¹ I just came across [Python]: PEP 370 -- Per user site-packages directory - Implementation (emphasis is mine):

The site module gets a new method adduserpackage() which adds the appropriate directory to the search path. The directory is not added if it doesn't exist when Python is started.

Installing python module within code

The officially recommended way to install packages from a script is by calling pip's command-line interface via a subprocess. Most other answers presented here are not supported by pip. Furthermore since pip v10, all code has been moved to pip._internal precisely in order to make it clear to users that programmatic use of pip is not allowed.

Use sys.executable to ensure that you will call the same pip associated with the current runtime.

import subprocess
import sys

def install(package):
    subprocess.check_call([sys.executable, "-m", "pip", "install", package])

How do you import a module right after installing it in the same file?

The answer posted by @Tom McLean did solve the issue, but I would like to point out the following:

The answer provided might throw errors for pip 10 or higher. If your version of pip is higher than 10, I recommend replacing

pip.main(['install', package])

with

os.system('pip install ' + package)

For more information on the errors that might be thrown if you are using pip10+, check out this link: https://github.com/pypa/pip/issues/5599

Cannot import dynamically installed python module

Well, this was a fast turnaround. I've been trying to figure this one out for a few days and finally came up with my answer a few minutes after asking on here.

If a path that does not yet exist is added to sys.path, it doesn't seem like it will ever be checked again when importing modules, even if it exists at a later point (or at least in python 2.7).

In my testing, site.USER_SITE didn't exist when I added it to sys.path. If I first made sure that that directory exists, then everything works how you would think it should:

import os
import pip
import site
import sys

# this makes it work
if not os.path.exists(site.USER_SITE):
    os.makedirs(site.USER_SITE)

# since I'm installing with --user, packages should be installed here,
# so make sure it's on the path
sys.path.insert(0, site.USER_SITE)

pip.main(["install", "--user", "package1"])
import package1

How do I automatically install missing python modules?

Installation issues are not subject of the source code!

You define your dependencies properly inside the setup.py of your package
using the install_requires configuration.

That's the way to go...installing something as a result of an ImportError
is kind of weird and scary. Don't do it.

How to load all modules in a folder?

List all python (.py) files in the current folder and put them as __all__ variable in __init__.py

from os.path import dirname, basename, isfile, join
import glob
modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')]

Install module before import

I fixed it in a different way. I used a module called "importlib". With this I was able to make a try and except way of trying to import the module and if it doesn't work, install it.

---

Related Topics