Namespaces Without Packages

Namespaces without packages

I’ve implemented a comprehensive solution and published it as a package, ‘box’.

Internally, ‘box’ modules uses an approach similar to packages; that is, it loads the code inside a dedicated namespace environment and then exports selected symbols into a module environment which is returned to the user, and optionally attached. The main difference to packages is that modules are more lightweight and easier to write (each R file is its own module), and can be nested.

Usage of the package is described in detail on its website.

Should one avoid a Python module being named in its own namespace?

Too long for a comment; TLDR: try to avoid extra imports and show the code to get concrete suggestions.

No, there is no established rationale for this. Do whatever makes sense. Having shorter imports is more user-friendly; cases when you have to introduce a no-op import layer on top are VERY RARE.

Note that from tqdm import tqdm actually imports object tqdm from module tqdm, so it is not really an empty import layer on top - the module contains a bunch of other objects.

Python project are diverse. In fact, many high profile projects are written in other languages, like C/C++. They all have different folder structure, so there is no one size fits all.

python namespaces vs packages: making a package the default namespace

You would add the 'old' names inside your new package by importing into the top-level package.

Names imported as globals in pypackage/__init__.py are attributes on the pypackage package. Make use of that to give access to 'legacy' locations:

# add all public names from pypackage.core.pypackage to the top level for
# legacy package use
from .core.pypackage import *

Now any code that uses import pypackage can use pypackage.foo and pypackage.bar if in reality these objects were defined in pypackage.core.pypackage instead.

Now, because pypackage is a setuptools namespace package you have a different problem; namespace packages are there for multiple separate distributions to install into so that top-level package must either be empty or only contain a minimum __init__.py file (namespace packages created with empty directories require Python 3.3).

If you are the only publisher of distributions that use this namespace, you can cheat a little here and use a single __init__.py file in your core package that could use pkg-util-style __init__.py file with the additional import I used above, but then you must not use any __init__.py files in other distribution packages or require that they all use the exact same __init__.py content. Coordination is key here.

Or you would have to use a different approach. Leave pypackage as a legacy wrapper module, and rename the new package format to use a new, different top-level name that can live next to the old module. At this point you can then just include the legacy package in your project, directly, as an extra top-level module.

Namespace vs regular package

Namespace packages

As of Python 3.3, we get namespace packages. These are a special kind of package that allows you to unify two packages with the same name at different points on your Python-path. For example, consider path1 and path2 as separate entries on your Python-path:

path1
+--namespace
   +--module1.py
   +--module2.py
path2
+--namespace
   +--module3.py
   +--module4.py

with this arrangement you should be able to do the following:

from namespace import module1, module3

thus you get the unification of two packages with the same name in a single namespace.

If either one of them gain an __init__.py that becomes the package - and you no longer get the unification as the other directory is ignored.

If both of them have an __init__.py, the first one in the PYTHONPATH (sys.path) is the one used.

`init.py` used to be required to make directory a package

Namespace packages are packages without the __init__.py.

For an example of a simple package, if you have a directory:

root
+--package
   +--file1.py
   +--file2.py
   ...

While you could run these files independently in the package directory, e.g. with python2 file1.py, under Python 2 you wouldn't be able to import the files as modules in the root directory, e.g.

import package.file1

would fail, and in order for it to work, you at least need this:

package
  +--__init__.py
  +--file1.py
  +--file2.py
  ...

__init__.py initializes the package so you can have code in the __init__.py that is run when the module is first imported:

run_initial_import_setup()

provide an __all__ list of names to be imported,

__all__ = ['star_import', 'only', 'these', 'names']

if the package is imported with the following:

from module import *

or you can leave the __init__.py completely empty if you only want to be able to import the remaining .py files in the directory.

Namespaces with `init.py` using pkgutil:

You could originally use pkgutil, available since Python 2.3. to accomplish adding namespaces, by adding the following into each separate package's __init__.py:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

Setuptools uses a similar method, again, all __init__.py files should contain the following (with no other code):

import pkg_resources
pkg_resources.declare_namespace(__name__)

Namespaces were more thoroughly addressed in PEP 420

See also more discussion on setuptools and Namespaces here:

http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages

working with package without Namespace in R

Based on the discussion above I am answering my own questions. All credit goes to who discussed this issue.

Create a file with the following text and put an extra line to avoid an error. Same in the directory for the package where you have description file.

exportPattern("^[^\\.]")

How do I create a namespace package in Python?

TL;DR:

On Python 3.3 you don't have to do anything, just don't put any __init__.py in your namespace package directories and it will just work. On pre-3.3, choose the pkgutil.extend_path() solution over the pkg_resources.declare_namespace() one, because it's future-proof and already compatible with implicit namespace packages.

Python 3.3 introduces implicit namespace packages, see PEP 420.

This means there are now three types of object that can be created by an import foo:

A module represented by a foo.py file
A regular package, represented by a directory foo containing an __init__.py file
A namespace package, represented by one or more directories foo without any __init__.py files

Packages are modules too, but here I mean "non-package module" when I say "module".

First it scans sys.path for a module or regular package. If it succeeds, it stops searching and creates and initalizes the module or package. If it found no module or regular package, but it found at least one directory, it creates and initializes a namespace package.

Modules and regular packages have __file__ set to the .py file they were created from. Regular and namespace packages have __path__set to the directory or directories they were created from.

When you do import foo.bar, the above search happens first for foo, then if a package was found, the search for bar is done with foo.__path__as the search path instead of sys.path. If foo.bar is found, foo and foo.bar are created and initialized.

So how do regular packages and namespace packages mix? Normally they don't, but the old pkgutil explicit namespace package method has been extended to include implicit namespace packages.

If you have an existing regular package that has an __init__.py like this:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

... the legacy behavior is to add any other regular packages on the searched path to its __path__. But in Python 3.3, it also adds namespace packages.

So you can have the following directory structure:

├── path1
│   └── package
│       ├── __init__.py
│       └── foo.py
├── path2
│   └── package
│       └── bar.py
└── path3
    └── package
        ├── __init__.py
        └── baz.py

... and as long as the two __init__.py have the extend_path lines (and path1, path2 and path3 are in your sys.path) import package.foo, import package.bar and import package.baz will all work.

pkg_resources.declare_namespace(__name__) has not been updated to include implicit namespace packages.

Namespaces Without Packages