How to Create a Namespace Package in Python

How do I create a namespace package in Python?

TL;DR:

On Python 3.3 you don't have to do anything, just don't put any __init__.py in your namespace package directories and it will just work. On pre-3.3, choose the pkgutil.extend_path() solution over the pkg_resources.declare_namespace() one, because it's future-proof and already compatible with implicit namespace packages.

Python 3.3 introduces implicit namespace packages, see PEP 420.

This means there are now three types of object that can be created by an import foo:

A module represented by a foo.py file
A regular package, represented by a directory foo containing an __init__.py file
A namespace package, represented by one or more directories foo without any __init__.py files

Packages are modules too, but here I mean "non-package module" when I say "module".

First it scans sys.path for a module or regular package. If it succeeds, it stops searching and creates and initalizes the module or package. If it found no module or regular package, but it found at least one directory, it creates and initializes a namespace package.

Modules and regular packages have __file__ set to the .py file they were created from. Regular and namespace packages have __path__set to the directory or directories they were created from.

When you do import foo.bar, the above search happens first for foo, then if a package was found, the search for bar is done with foo.__path__as the search path instead of sys.path. If foo.bar is found, foo and foo.bar are created and initialized.

So how do regular packages and namespace packages mix? Normally they don't, but the old pkgutil explicit namespace package method has been extended to include implicit namespace packages.

If you have an existing regular package that has an __init__.py like this:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

... the legacy behavior is to add any other regular packages on the searched path to its __path__. But in Python 3.3, it also adds namespace packages.

So you can have the following directory structure:

├── path1
│   └── package
│       ├── __init__.py
│       └── foo.py
├── path2
│   └── package
│       └── bar.py
└── path3
    └── package
        ├── __init__.py
        └── baz.py

... and as long as the two __init__.py have the extend_path lines (and path1, path2 and path3 are in your sys.path) import package.foo, import package.bar and import package.baz will all work.

pkg_resources.declare_namespace(__name__) has not been updated to include implicit namespace packages.

Python Namespace Packages in Python3

Late to the party, but never hurts to help fellow travellers down the namespace path in Python!

#1:

With the __init__.py, which of these should I be using (if any)?:

It depends, There are three ways to do namespace packages as listed here:

Use native namespace packages. This type of namespace package is defined in PEP 420 and is available in Python 3.3 and later. This is recommended if packages in your namespace only ever need to support Python 3 and installation via pip.
Use pkgutil-style namespace packages. This is recommended for new packages that need to support Python 2 and 3 and installation via both pip and python setup.py install.
Use pkg_resources-style namespace packages. This method is recommended if you need compatibility with packages already using this method or if your package needs to be zip-safe.

If you are using #2 (pkgutil-style) or #3 (pkg_resources-style), then you will have to use the corresponding style for __init__.py files. If you use native namespaces then no __init__.py in the namespace directory.

#2:

With setup.py, do I still need to add the namespace_modules parameter, and if so, would I use namespace_modules=['org.common'], or namespace_modules=['org', 'common']?

If your choice of namespace package is not native style, then yes, you will need namespace_packages in your setup().

#3:

Could I forgo all of the above by just implementing this differently somehow? Perhaps something simpler or more "pythonic"?

Since you ended up down to a complex topic in python, it seems you know what you are doing, what you want and identified that creating a Python Namespace package is the way to do it. This would be considered a pythonic way to solve a problem.

Adding to your questions, here are a few things I discovered:

I read PEP420, the Python Packaging guide and spent a lot of time understanding the namespace packages, and I generally understood how it worked. I read through a couple of answers here, here, here, and this thread on SO as well - the example here and on the Git link shared by Rob.

My problem however was after I created my package. As all the instructions and sample code explicitly listed the package in the setuptools.setup(package=[]) function, my code failed. My sub-packages/directories were not included. Digging deeper, I found out that setuptools has a find_namespace_package() function that helps in adding sub-packages too

EDIT:

Link to find_namespace_packages() (setuptools version greater than 40.1.0): https://setuptools.readthedocs.io/en/latest/setuptools.html#find-namespace-packages

EDIT (08/09/2019):

To complete the answer, let me also restructure with an example.

The following solution is assuming Python 3.3+ which has support for implicit namespace packages

Since you are looking for a solution for Python version 3.5 or later, let's take the code samples provided and elaborate further.

Let's assume the following:

Namespace/Python package name : org

Distribution packages: org_client, org_common

Python: 3.3+

setuptools: 40.1.0

For you to do the following

from org.client.client1 import mod1
from org.common import config

And keeping your top level directories the same, viz. org_client_client1_mod1 and org_common_config, you can change your structure to the following

Repository 1:

org_client_client1_mod1/
  setup.py
  org/
    client/
      client1/
        __init__.py
        submod1/
          __init__.py
        mod1/
          __init__.py
          somefile.py
        file1.py

Updated setup.py

from setuptools import find_namespace_packages, setup
setup(
    name="org_client",
    ...
    packages=find_namespace_packages(), # Follows similar lookup as find_packages()
    ...
)

Repository 2:

org_common_config/
  setup.py
  org/
    common/
      __init__.py
      config/
        __init__.py
        someotherfile.py

Updated setup.py:

from setuptools import find_namespace_packages, setup
setup(
    name="org_common",
    ...
    packages=find_namespace_packages(), # Follows similar lookup as find_packages()
    ...
)

To install (using pip):

(venv) $ pip3 install org_common_config/
(venv) $ pip3 install org_client_client1_mod1/

Updated pip list will show the following:

(venv) $ pip3 list
...
org_client
org_common
...

But they won't be importable, for importing you will have to follow org.client and org.common notation.

To understand why, you can browse here (assuming inside venv):

(venv) $ cd venv/lib/python3.5/site-packages/
(venv) $ ls -l | grep org

You'll see that there's no org_client or org_common directories, they are interpreted as a namespace package.

(venv) $ cd venv/lib/python3.5/site-packages/org/
(venv) $ ls -l
client/
common/
...

How to create nested namespace packages for setuptools distribution

Yes there is more than one way. Please read section "Nested namespace packages" in PEP 420.

In python >= 3.3, the easiest way to make nested namespace is to delete (do not include) file __init__.py in the specific folders ("lvl1" and "lvl2") in every distributable parts. In each of the setup.py, explicitly list all the packages in the deepest namespace.

"lvl1_part1/setup.py"

setup(
    name='lvl1_part1',
    ...
    zip_safe=False,
    packages=['lvl1.lvl2.sub1']
)

"lvl1_part2/setup.py"

setup(
    name='lvl1_part2',
    ...
    zip_safe=False,
    packages=['lvl1.lvl2.sub2']
)

The file structure for testing:

lvl1_part1/
           setup.py
           lvl1/
                lvl2/
                     sub1/
                          __init__.py
lvl1_part2/
           setup.py
           lvl1/
                lvl2/
                     sub2/
                          __init__.py

To make the above packages compatible to older python versions, please add the pkgutil magic file to each of the "lvl1" and "lvl2" folders.

Credits: The example above is modified from https://github.com/pypa/sample-namespace-packages/tree/master/pkgutil

Create name_space package that contains standalone installable sub-packages

As pointed out by @sinoroc:

When using package_dir, you can"t really use find_namespace_packages, you have to either write the list manually or modify it before assigning it to packages.

As a result to achieve the desired behaviour I had to modify the packages list before supplying it to the setuptools.setup method. This could be done in several ways.

1. Add a virtual shortened package to the packages list

We can add extra (shortened) module entries for each of the redundant folders to the packages list. This can be done by using the following setup.py file:

setup.py file

from setuptools import setup, find_namespace_packages

# Retrieve package list
PACKAGES = find_namespace_packages(include=["meta_package*"])+["meta_package.installable_subpackage"]

setup(
    name="meta_package",
    ...
    packages=PACKAGES,
    package_dir={
        "meta_package.installable_subpackage": "meta_package/installable_subpackage/installable_subpackage",
    },
)

To do this automatically for a number of sub-packages, you can use the following code:

# Retrieve package list
PACKAGES = find_namespace_packages(include=["meta_package*"])

# Add extra virtual shortened package for each of namespace_pkgs that contain redundant folders
namespace_pkgs = ["installable_subpackage"]
exclusions = r"|".join(
    [r"\." + item + r"\.(?=" + item + r".)" for item in namespace_pkgs]
)
PACKAGE_DIR = {}
for package in PACKAGES:
    sub_tmp = re.sub(exclusions, ".", package)
    if sub_tmp is not package:
        PACKAGE_DIR[sub_tmp] = package.replace(".", "/")
PACKAGES.extend(PACKAGE_DIR.keys())

setup(
    name="meta_package",
    ...
    packages=PACKAGES,
    package_dir=PACKAGE_DIR,
)

2. Modify package list such that it only contains the shortened package

Alternatively, if you want to replace the long module name with a shorter name fully, you can use the following setup.py:

setup.py file

from setuptools import setup, find_namespace_packages

# Retrieve package list
PACKAGES = find_namespace_packages(include=["meta_package*"])

# Remove redundant folders from the package list
PACKAGES = [re.sub(r"\.installable_subpackage\.(?=installable_subpackage.)", ".", package) for package in PACKAGES]

setup(
    name="meta_package",
    ...
    packages=PACKAGES,
    package_dir={
        "meta_package.installable_subpackage": "meta_package/installable_subpackage/installable_subpackage",
    },
)

This can also be done automatically for a number of sub-packages using the following code:

# Remove redundant folder from package list
red_folders = ["installable_subpackage"]
exclusions = r"|".join(
    [r"\." + item + r"\.(?=" + item + r".)" for item in red_folders]
)
PACKAGE_DIR = {}
for index, package in enumerate(PACKAGES):
    sub_tmp = re.sub(exclusions, ".", package)
    if sub_tmp is not package:
        PACKAGES[index] = sub_tmp

IMPORTANT REMARKS

While using the methods described above, it is important to note that they do not yet work if a package is installed in development mode (see this issue). It is, therefore, better to use the first method since developers can still use the long module name while users can also use the shorter module name.
Please remember that the methods described above don't work if there is an __init__.py file in your namespace package root folder (see the setuputils documentation).

Update

After I got this answer, I tried to put this logic into a setup.cfg file so that it would be PEP517/518 compatible. While doing this, I ran into some problems. The solution to this can be found on this issue I created on the setuptools GitHub page. An example repository can be found here.

How to install multiple python namespace packages in editable mode

Both of your modules are in the regular package dende.api, because they both contain an __init__.py. So, they are not really in an implicit namespace package. Only one of the two dende.apis will be available.

It works when installing them in non-editable mode only by accident, because the files are written into the same directory on disk, but not because of the namespace package mechanism.

You can repair that by removing the __init__.pys and thereby making dende.api a namespace package or by renaming the two apis into distinct packages github-api and gitlab-api, which then both are in the namespace package dende.

A more detailed explanation of the namespace package mechanisms for a somehow related problem (why it makes only little sense to have namespace packages inside regular ones) can be found here: https://stackoverflow.com/a/62992832

python namespaces vs packages: making a package the default namespace

You would add the 'old' names inside your new package by importing into the top-level package.

Names imported as globals in pypackage/__init__.py are attributes on the pypackage package. Make use of that to give access to 'legacy' locations:

# add all public names from pypackage.core.pypackage to the top level for
# legacy package use
from .core.pypackage import *

Now any code that uses import pypackage can use pypackage.foo and pypackage.bar if in reality these objects were defined in pypackage.core.pypackage instead.

Now, because pypackage is a setuptools namespace package you have a different problem; namespace packages are there for multiple separate distributions to install into so that top-level package must either be empty or only contain a minimum __init__.py file (namespace packages created with empty directories require Python 3.3).

If you are the only publisher of distributions that use this namespace, you can cheat a little here and use a single __init__.py file in your core package that could use pkg-util-style __init__.py file with the additional import I used above, but then you must not use any __init__.py files in other distribution packages or require that they all use the exact same __init__.py content. Coordination is key here.

Or you would have to use a different approach. Leave pypackage as a legacy wrapper module, and rename the new package format to use a new, different top-level name that can live next to the old module. At this point you can then just include the legacy package in your project, directly, as an extra top-level module.

Packaging and building python namespace packages

If you want to use latest python native namespace packages to allow users to install extensions-like packages (mypkg-core, mypkg-compute,...), you should create different folders for each package like so

mypkg-core/
    mypkg/ # no __init__.py here
        core/
            __init__.py
        setup.py

mypkg-compute/
    mypkg/ # no __init__.py here
        compute/
            __init__.py
        setup.py

where each setup.py is like the following

from setuptools import setup, find_namespace_packages

setup(
    name='mypkg-core',
    ...
    packages=find_namespace_packages(include=['mypkg.*'])
)

And you will need to build a whl for each package.

With this configuration, users will be able to

pip install mypkg-core
pip install mypkg-compute

And access them via the same mypkg namespace:

import mypkg.core
import mypkg.compute

Don't forget to use find_namespace_package from setuptools in setup.py for each subpkg.

Please always refer to the official documentation about native namespace packages..

However, I would suggest an easier way to distribute your package by adding the subpackages dependencies as extras grouped with the name of the subpackage. This way the users can use the subpackage only if they install the whole package with

pip install mypkg[compute]

Current state of python namespace packages

The latest version of Python which is Python 3.7 uses the native namespace packages approach to create namespace packages which are defined in PEP 420.

There are currently three different approaches to creating namespace packages:

Use native namespace packages. This type of namespace package is defined in PEP 420 and is available in Python 3.3 and later. This is recommended if packages in your namespace only ever need to support Python 3 and installation via pip.
Use pkgutil-style namespace packages. This is recommended for new packages that need to support Python 2 and 3 and installation via both pip and python setup.py install.
Use pkg_resources-style namespace packages. This method is recommended if you need compatibility with packages already using this method or if your package needs to be zip-safe.

Reference: Packaging namespace packages