Circular Import Dependency in Python

Circular dependency in Python

Importing Python Modules is a great article that explains circular imports in Python.

The easiest way to fix this is to move the path import to the end of the node module.

What happens when using mutual or circular (cyclic) imports in Python?

There was a really good discussion on this over at comp.lang.python last year. It answers your question pretty thoroughly.

Imports are pretty straightforward really. Just remember the following:

'import' and 'from xxx import yyy' are executable statements. They execute
when the running program reaches that line.

If a module is not in sys.modules, then an import creates the new module
entry in sys.modules and then executes the code in the module. It does not
return control to the calling module until the execution has completed.

If a module does exist in sys.modules then an import simply returns that
module whether or not it has completed executing. That is the reason why
cyclic imports may return modules which appear to be partly empty.

Finally, the executing script runs in a module named __main__, importing
the script under its own name will create a new module unrelated to
__main__.

Take that lot together and you shouldn't get any surprises when importing
modules.

How to avoid circular imports in Python?

Only import the module, don't import from the module:

Consider a.py:

import b

class A:
    def bar(self):
        return b.B()

and b.py:

import a

class B:
    def bar(self):
        return a.A()

This works perfectly fine.

Circular import dependency in Python

If a depends on c and c depends on a, aren't they actually the same unit then?

You should really examine why you have split a and c into two packages, because either you have some code you should split off into another package (to make them both depend on that new package, but not each other), or you should merge them into one package.

Why do circular imports seemingly work further up in the call stack but then raise an ImportError further down?

I think the answer by jpmc26, while by no means wrong, comes down too heavily on circular imports. They can work just fine, if you set them up correctly.

The easiest way to do so is to use import my_module syntax, rather than from my_module import some_object. The former will almost always work, even if my_module included imports us back. The latter only works if my_object is already defined in my_module, which in a circular import may not be the case.

To be specific to your case: Try changing entities/post.py to do import physics and then refer to physics.PostBody rather than just PostBody directly. Similarly, change physics.py to do import entities.post and then use entities.post.Post rather than just Post.

Understanding behavior of Python imports and circular dependencies

When Python starts loading the pkg.a module, it sets sys.modules['pkg.a'] to the corresponding module object, but it only sets the a attribute of the pkg module object at the very end of loading the pkg.a module. This will be relevant later.

---

Relative imports are from imports, and they behave the same. After from . import whatever figures out that . refers to the pkg package, it goes ahead with the regular from pkg import whatever logic.

When c.py hits from . import a, first, it sees that pkg.a is already in sys.modules, indicating that pkg.a has already been loaded or is in the middle of being loaded. (It's in the middle of being loaded, but this code path doesn't care.) It skips to the second part of its job, retrieving pkg.a and assigning it to the a name in the local namespace, but it doesn't just retrieve sys.modules['pkg.a'] to do this.

You know how you can do stuff like from os import open, even though os.open is a function, not a module? That kind of import can't go through sys.modules['os.open'], because os.open isn't a module and isn't in sys.modules. Instead, all from imports, including all relative imports, attempt an attribute lookup on the module they're importing names from. from . import a looks up the a attribute on the pkg module object, but it's not there, because that attribute only gets set when pkg.a finishes loading.

On Python 2, that's it. End of import. ImportError here. On Python 3 (specifically 3.5+), because they wanted to encourage relative imports and this behavior is really inconvenient, from imports try one more step. If the attribute lookup fails, now they try sys.modules. pkg.a is in sys.modules, so the import succeeds. You can see the discussion for this change in the CPython issue tracker at issue 17636.

Python circular import in custom package and __init__.py

TL;DR: replace from . import Query with from .elastic_query import Query

Explanation:

When you import something from libs.elastic_search_hunt module it loads __init__.py at first. Since every module executes at first import __init__.py also being executed.

Then Python executes code from __init__.py and at second line

from libs.elastic_search_hunt.search_processor import SearchProcessor

it imports search_processor.py. Since it's first import - file must be executed - therefore all your imports in that file must be executed right now as well:

As you mentioned you have the following imports in your file:

from . import Query
from . import Result

At this point you tell python to load libs.elastic_search_hunt entire module and take Query, Result from it. So Python does.

It makes an attempt to load libs/elastic_search_hunt/__init__.py but wait... it is still not loaded completely. So it must load it, but in order to load it properly it must firstly load search_processor which requires elastic_search_hunt/__init__.py to be loaded.... oh well, there's a loop.

So in order to avoid such behaviour you should explicitly say from which module exactly you wish to load Query and Result, therefore change

from . import Query
from . import Result

to

from .elastic_query import Query
from .elastic_query_result import Result

Example: Failed
Example: Success

Avoiding circular imports with type annotations in situations where __future__.annotations is insufficient

In the most cases using typing.TYPE_CHECKING helps to resolve circular import issues.

# a.py
from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from b import B

class A: pass
def foo(b: B) -> None: pass

# b.py
from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from a import A

class B: pass
def bar(a: A) -> None: pass

# __main__.py
from a import A
from b import B

However, for exactly your MRE it won't work. If the circular dependency is introduced not only by type annotations (e.g. your type aliases), the resolving may become really tricky.

If you don't need Foo available at runtime in your example, it can be declared in if TYPE_CHECKING: block too, mypy will interpret that properly. If it is for runtime too, then everything depends on exact code structure (in your MRE dropping import b is enough). Union type can be declared in separate file that imports a, b and c and creates Union. If you need this union in a, b or c, then things are a bit more complicated, probably some functionality needs to be extracted into separate file d that creates union and uses it (also the code will be a bit cleaner this way, because every file will contain only common functionality).

---

Related Topics