List All the Modules That Are Part of a Python Package

List all the modules that are part of a python package?

Yes, you want something based on pkgutil or similar -- this way you can treat all packages alike regardless if they are in eggs or zips or so (where os.listdir won't help).

import pkgutil

# this is the package we are inspecting -- for example 'email' from stdlib
import email

package = email
for importer, modname, ispkg in pkgutil.iter_modules(package.__path__):
    print "Found submodule %s (is a package: %s)" % (modname, ispkg)

How to import them too? You can just use __import__ as normal:

import pkgutil

# this is the package we are inspecting -- for example 'email' from stdlib
import email

package = email
prefix = package.__name__ + "."
for importer, modname, ispkg in pkgutil.iter_modules(package.__path__, prefix):
    print "Found submodule %s (is a package: %s)" % (modname, ispkg)
    module = __import__(modname, fromlist="dummy")
    print "Imported", module

Is there a standard way to list names of Python modules in a package?

Maybe this will do what you're looking for?

import imp
import os
MODULE_EXTENSIONS = ('.py', '.pyc', '.pyo')

def package_contents(package_name):
    file, pathname, description = imp.find_module(package_name)
    if file:
        raise ImportError('Not a package: %r', package_name)
    # Use a set because some may be both source and compiled.
    return set([os.path.splitext(module)[0]
        for module in os.listdir(pathname)
        if module.endswith(MODULE_EXTENSIONS)])

How to list all functions in a module?

Use the inspect module:

from inspect import getmembers, isfunction

from somemodule import foo
print(getmembers(foo, isfunction))

Also see the pydoc module, the help() function in the interactive interpreter and the pydoc command-line tool which generates the documentation you are after. You can just give them the class you wish to see the documentation of. They can also generate, for instance, HTML output and write it to disk.

How can I list all packages/modules available to Python from within a Python script?

Alright, I was curious, and I digged a bit into pkgutil, and I came up with this, which is much simpler than I expected:

list(pkgutil.iter_modules())

It lists all top-level packages/modules available either as regular files or zip packages, without loading them. It will not see other types of packages though, unless they properly register with the pkgutil internals.

Each returned entry is a 3-tuple with the items:

1. module_finder: The file finder instance that found the module
2. name: The name of the module
3. ispkg: A boolean specifying whether it is a regular module or a package.

Example 3-tuple:

(FileFinder('/usr/lib/python3/dist-packages'), 'PIL', True)

And I can confirm that this did not load the PIL package:

In [11]: sys.modules['PIL']
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-11-b0fc0af6cc34> in <module>()
----> 1 sys.modules['PIL']

KeyError: 'PIL'

How do I get a list of locally installed Python modules?

Solution

Do not use with pip > 10.0!

My 50 cents for getting a pip freeze-like list from a Python script:

import pip
installed_packages = pip.get_installed_distributions()
installed_packages_list = sorted(["%s==%s" % (i.key, i.version)
     for i in installed_packages])
print(installed_packages_list)

As a (too long) one liner:

sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])

Giving:

['behave==1.2.4', 'enum34==1.0', 'flask==0.10.1', 'itsdangerous==0.24', 
 'jinja2==2.7.2', 'jsonschema==2.3.0', 'markupsafe==0.23', 'nose==1.3.3', 
 'parse-type==0.3.4', 'parse==1.6.4', 'prettytable==0.7.2', 'requests==2.3.0',
 'six==1.6.1', 'vioozer-metadata==0.1', 'vioozer-users-server==0.1', 
 'werkzeug==0.9.4']

Scope

This solution applies to the system scope or to a virtual environment scope, and covers packages installed by setuptools, pip and (god forbid) easy_install.

My use case

I added the result of this call to my flask server, so when I call it with http://example.com/exampleServer/environment I get the list of packages installed on the server's virtualenv. It makes debugging a whole lot easier.

Caveats

I have noticed a strange behaviour of this technique - when the Python interpreter is invoked in the same directory as a setup.py file, it does not list the package installed by setup.py.

Steps to reproduce:

Create a virtual environment

$ cd /tmp
$ virtualenv test_env
New python executable in test_env/bin/python
Installing setuptools, pip...done.
$ source test_env/bin/activate
(test_env) $ 

Clone a git repo with setup.py

(test_env) $ git clone https://github.com/behave/behave.git
Cloning into 'behave'...
remote: Reusing existing pack: 4350, done.
remote: Total 4350 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (4350/4350), 1.85 MiB | 418.00 KiB/s, done.
Resolving deltas: 100% (2388/2388), done.
Checking connectivity... done.

We have behave's setup.py in /tmp/behave:

(test_env) $ ls /tmp/behave/setup.py
/tmp/behave/setup.py

Install the python package from the git repo

(test_env) $ cd /tmp/behave && pip install . 
running install
...
Installed /private/tmp/test_env/lib/python2.7/site-packages/enum34-1.0-py2.7.egg
Finished processing dependencies for behave==1.2.5a1

If we run the aforementioned solution from /tmp

>>> import pip
>>> sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])
['behave==1.2.5a1', 'enum34==1.0', 'parse-type==0.3.4', 'parse==1.6.4', 'six==1.6.1']
>>> import os
>>> os.getcwd()
'/private/tmp'

If we run the aforementioned solution from /tmp/behave

>>> import pip
>>> sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])
['enum34==1.0', 'parse-type==0.3.4', 'parse==1.6.4', 'six==1.6.1']
>>> import os
>>> os.getcwd()
'/private/tmp/behave'

behave==1.2.5a1 is missing from the second example, because the working directory contains behave's setup.py file.

I could not find any reference to this issue in the documentation. Perhaps I shall open a bug for it.

How to get a list of all the Python standard library modules?

Why not work out what's part of the standard library yourself?

import distutils.sysconfig as sysconfig
import os
std_lib = sysconfig.get_python_lib(standard_lib=True)
for top, dirs, files in os.walk(std_lib):
    for nm in files:
        if nm != '__init__.py' and nm[-3:] == '.py':
            print os.path.join(top, nm)[len(std_lib)+1:-3].replace(os.sep, '.')

gives

abc
aifc
antigravity
--- a bunch of other files ----
xml.parsers.expat
xml.sax.expatreader
xml.sax.handler
xml.sax.saxutils
xml.sax.xmlreader
xml.sax._exceptions

Edit: You'll probably want to add a check to avoid site-packages if you need to avoid non-standard library modules.

---

Related Topics