_Getattr_ on a Module

__getattr__ on a module

A while ago, Guido declared that all special method lookups on
new-style classes bypass __getattr__ and __getattribute__. Dunder methods had previously worked on modules - you could, for example, use a module as a context manager simply by defining __enter__ and __exit__, before those tricks broke.

Recently some historical features have made a comeback, the module __getattr__ among them, and so the existing hack (a module replacing itself with a class in sys.modules at import time) should be no longer necessary.

In Python 3.7+, you just use the one obvious way. To customize attribute access on a module, define a __getattr__ function at the module level which should accept one argument (name of attribute), and return the computed value or raise an AttributeError:

# my_module.py

def __getattr__(name: str) -> Any:
    ...

This will also allow hooks into "from" imports, i.e. you can return dynamically generated objects for statements such as from my_module import whatever.

On a related note, along with the module getattr you may also define a __dir__ function at module level to respond to dir(my_module). See PEP 562 for details.

__getattr__ for module running twice on import?

Your __getattr__ returns None for all attributes, including __path__, and if a module has a __path__ attribute, it's treated as a package.

For a package, from test import x, y needs to handle possible submodules named test.x and test.y. The code that does this handling uses hasattr first, to test whether test already has x and y attributes:

elif not hasattr(module, x):
    ...

and that hasattr is responsible for the first __getattr__('x') and __getattr__('y') calls.

The second __getattr__('x') and __getattr__('y') calls are just the ones you'd expect to happen, to retrieve test.x and test.y for the import.

---

Your __getattr__ shouldn't return a value for special attributes like __path__.

Also, unrelated, naming a module test is a bad idea, because the standard library already claimed that name for Python's own test suite.

How does one use python's __import__() and getattr() to correctly to instantiate nested classes?

Two ways to do this, suppose you have a string representing attribute access and a nested object:

>>> from types import SimpleNamespace
>>> module = SimpleNamespace(foo=SimpleNamespace(bar=SimpleNamespace(baz='tada!')))
>>> module
namespace(foo=namespace(bar=namespace(baz='tada!')))

The first is to basically parse the string yourself by splitting and using getattr in a loop (or even, reduce!):

>>> from functools import reduce
>>> reduce(getattr, "foo.bar.baz".split('.'), module)
'tada!'

Which is just equivalent to:

>>> result = module
>>> for attr in "foo.bar.baz".split("."):
...     result = getattr(result, attr)
...
>>> result
'tada!'

Or use the built-in functools.attrgetter factory function as a one-off:

>>> import operator
>>> operator.attrgetter("foo.bar.baz")(module)
'tada!'

Python __getattr__ executed multiple times

TL;DR the first "test" printed is a side-effect of the "from import" implementation, i.e. it's printed during creation of lib module. The second "test" is from subsequent access of dynamic attribute on the module directly.

Knowing that importlib is implemented in Python code, modify your lib.py slightly to also dump a trace:

# lib.py
from traceback import print_stack

def __getattr__(name):
    print_stack()
    print(name)
    print("-" * 80)

This gives the hint to pinpoint the library location in importlib which triggers double attribute access:

$ python3 main.py 
  File "main.py", line 3, in <module>
    from lib import test
  File "<frozen importlib._bootstrap>", line 1019, in _handle_fromlist
  File "/private/tmp/lib.py", line 5, in __getattr__
    print_stack()
__path__
--------------------------------------------------------------------------------
  File "main.py", line 3, in <module>
    from lib import test
  File "<frozen importlib._bootstrap>", line 1032, in _handle_fromlist
  File "/private/tmp/lib.py", line 5, in __getattr__
    print_stack()
test
--------------------------------------------------------------------------------
  File "main.py", line 3, in <module>
    from lib import test
  File "/private/tmp/lib.py", line 5, in __getattr__
    print_stack()
test
--------------------------------------------------------------------------------

Now we can find the answer easily by RTFS (below I use Python v3.7.6, switch on git to the exact tag you use in case of different version). Look in importlib._bootstrap. _handle_fromlist at the indicated line numbers.

_handle_fromlist is a helper intended to load package submodules in a from import. Step 1 is to see if the module is a package at all:

if hasattr(module, '__path__'):

The __path__ access comes there, on line 1019. Because your __getattr__ returns None for all inputs, hasattr returns True here, so your module looks like a package, and the code continues on. (If hasattr had returned False, _handle_fromlist would abort at this point.)

The "fromlist" here will have the name you requested, ["test"], so we go into the for-loop with x="test" and on line 1032 there is the "extra" invocation:

elif not hasattr(module, x):

from lib import test will only attempt to load a lib.test submodule if lib does not already have a test attribute. This check is testing whether the attribute exists, to see if _handle_fromlist needs to attempt to load a submodule.

Should you return different values for the first and second invocation of __getattr__ with name "test", then the second value returned is the one which will actually be received within main.py.

__getattr__ within a module in python

You need to look for the name as a string; and I'd use hasattr() here to test for that name:

if hasattr(my_module, 'exists_method'):
    print 'Method found!"

This works if my_module.exists_method exists, but not if you run this code inside my_module.

If exists_method is contained in the current module, you would need to use globals() to test for it:

if 'exists_method' in globals():
    print 'Method found!'

Using getattr() to access built-in functions

The builtins module:

You could try importing builtins module:

>>> import builtins
>>> getattr(builtins, 'abs')
<built-in function abs>
>>> 

As mentioned in the documentation:

This module provides direct access to all ‘built-in’ identifiers of Python; for example, builtins.open is the full name for the built-in function open(). See Built-in Functions and Built-in Constants for documentation.

So the above mentions that builtins.open is the open function. So abs is the same builtins.abs is the same thing as abs. But for gettatr, getattr(builtins, 'abs') is also the same as builtins.abs.

The original __bultins__ (NOT RECOMMENDED):

You could try getting from the __builtins__:

>>> getattr(__builtins__, 'abs')
<built-in function abs>
>>> 

As mentioned in the documentation:

CPython implementation detail: Users should not touch __builtins__; it
is strictly an implementation detail. Users wanting to override values
in the builtins namespace should import the builtins module and modify
its attributes appropriately.

The builtins namespace associated with the execution of a code block
is actually found by looking up the name __builtins__ in its global
namespace; this should be a dictionary or a module (in the latter case
the module’s dictionary is used). By default, when in the main
module, __builtins__ is the built-in module builtins; when in any
other module, __builtins__ is an alias for the dictionary of the
builtins module itself.

As you can see, it's not recommended, also usually the __builtins__ is a dict, rather than a module.

If you write your code in modules, __builtins__ would return dictionary aliases of the builtins module, which would give something like: {..., 'abs': <built-in function abs>, ...}.

More on getattr:

Just to have more idea about getattr, as mentioned in the documentation:

Return the value of the named attribute of object. name must be a string. If the string is the name of one of the object’s attributes, the result is the value of that attribute. For example, getattr(x, 'foobar') is equivalent to x.foobar. If the named attribute does not exist, default is returned if provided, otherwise AttributeError is raised.

So:

>>> import builtins
>>> getattr(builtins, 'abs')
<built-in function abs>
>>> 

Is the same as:

>>> import builtins
>>> builtins.abs
<built-in function abs>
>>> 

So you might be wondering since:

>>> abs
<built-in function abs>

Gives the same thing, why we can't just do:

getattr(abs)

The reason we can't do that is that that getattr is suppose to be calling methods/functions/classes of a classes/modules.

The reason using getattr(builtins, 'abs') works is because builtins is a module and abs is a class/method, it stores all the built-in Python keywords as methods in that module.

All the keywords are showed on this page of the documentation.

---

Related Topics