How to Intercept Calls to Python's "Magic" Methods in New Style Classes

How can I intercept calls to python's magic methods in new style classes?

For performance reasons, Python always looks in the class (and parent classes') __dict__ for magic methods and does not use the normal attribute lookup mechanism. A workaround is to use a metaclass to automatically add proxies for magic methods at the time of class creation; I've used this technique to avoid having to write boilerplate call-through methods for wrapper classes, for example.

class Wrapper(object):
    """Wrapper class that provides proxy access to an instance of some
       internal instance."""

    __wraps__  = None
    __ignore__ = "class mro new init setattr getattr getattribute"

    def __init__(self, obj):
        if self.__wraps__ is None:
            raise TypeError("base class Wrapper may not be instantiated")
        elif isinstance(obj, self.__wraps__):
            self._obj = obj
        else:
            raise ValueError("wrapped object must be of %s" % self.__wraps__)

    # provide proxy access to regular attributes of wrapped object
    def __getattr__(self, name):
        return getattr(self._obj, name)

    # create proxies for wrapped object's double-underscore attributes
    class __metaclass__(type):
        def __init__(cls, name, bases, dct):

            def make_proxy(name):
                def proxy(self, *args):
                    return getattr(self._obj, name)
                return proxy

            type.__init__(cls, name, bases, dct)
            if cls.__wraps__:
                ignore = set("__%s__" % n for n in cls.__ignore__.split())
                for name in dir(cls.__wraps__):
                    if name.startswith("__"):
                        if name not in ignore and name not in dct:
                            setattr(cls, name, property(make_proxy(name)))

Usage:

class DictWrapper(Wrapper):
    __wraps__ = dict

wrapped_dict = DictWrapper(dict(a=1, b=2, c=3))

# make sure it worked....
assert "b" in wrapped_dict                        # __contains__
assert wrapped_dict == dict(a=1, b=2, c=3)        # __eq__
assert "'a': 1" in str(wrapped_dict)              # __str__
assert wrapped_dict.__doc__.startswith("dict()")  # __doc__

Intercept magic method calls in python class

Talking about "passing by reference" will only confuse you. Keep that terminology to languages where you can have a choice on that, and where it makes a difference. In Python you always pass objects around - and this passing is the equivalent of "passing by reference" - for all objects - from None to int to a live asyncio network connection pool instance.

With that out of the way: the algorithm the language follows to retrieve attributes from an object is complicated, have details - implementing __getattr__ is just the tip of the iceberg. Reading the document called "Data Model" in its entirety will give you a better grasp of all the mechanisms involved in retrieving attributes.

That said, here is how it works for "magic" or "dunder" methods - (special functions with two underscores before and two after the name): when you use an operator that requires the existence of the method that implements it (like __add__ for +), the language checks the class of your object for the __add__ method - not the instance. And __getattr__ on the class can dynamically create attributes for instances of that class only.
But that is not the only problem: you could create a metaclass (inheriting from type) and put a __getattr__ method on this metaclass. For all querying you would do from Python, it would look like your object had the __add__ (or any other dunder method) in its class. However, for dunder methods, Python do not go through the normal attribute lookup mechanism - it "looks" directly at the class, if the dunder method is "physically" there. There are slots in the memory structure that holds the classes for each of the possible dunder methods - and they either refer to the corresponding method, or are "null" (this is "viewable" when coding in C on the Python side, the default dir will show these methods when they exist, or omit them if not). If they are not there, Python will just "say" the object does not implement that operation and period.

The way to work around that with a proxy object like you want is to create a proxy class that either features the dunder methods from the class you want to wrap, or features all possible methods, and upon being called, check if the underlying object actually implements the called method.

That is why "serious" code will rarely, if ever, offer true "transparent" proxy objects. There are exceptions, but from "Weakrefs", to "super()", to concurrent.futures, just to mention a few in the core language and stdlib, no one attempts a "fully working transparent proxy" - instead, the api is more like you call a ".value()" or ".result()" method on the wrapper to get to the original object itself.

However, it can be done, as I described above. I even have a small (long unmaintained) package on pypi that does that, wrapping a proxy for a future.
The code is at https://bitbucket.org/jsbueno/lelo/src/master/lelo/_lelo.py

Python inherit magic methods based off __init__ value

An easy way would be to use a proxy class, for example wrapt.ObjectProxy. It will behave exactly like the "proxied" class except for the overridden methods. However instead of self._obj you can simply use self.__wrapped__ to access the "unproxied" object.

from wrapt import ObjectProxy

class Wrapper(ObjectProxy):
    def __getattr__(self, name):
        print('getattr')
        return getattr(self.__wrapped__, name)

    def __getitem__(self, key):
        print('getitem')
        return self.__wrapped__[key]

    def __setitem__(self, key, val):
        print('setitem')
        self.__wrapped__[key] = val

    def __repr__(self):
        return repr(self.__wrapped__)

This behaves like a dict if you wrap a dict:

>>> d = Wrapper({'foo': 10})
>>> d['foo']
getitem
10
>>> 'foo' in d   # "inherits" __contains__
True

and like a list, if a list is wrapped:

>>> d = Wrapper([1,2,3])
>>> d[0]
getitem
1
>>> for i in d:   # "inherits" __iter__
...     print(i)
1
2
3

How to intercept a method call which doesn't exist?

Overwrite the __getattr__() magic method:

class MagicClass(object):
    def __getattr__(self, name):
        def wrapper(*args, **kwargs):
            print "'%s' was called" % name
        return wrapper

ob = MagicClass()
ob.unknown_method()
ob.unknown_method2()

prints

'unknown_method' was called
'unknown_method2' was called

Dynamically inherit all Python magic methods from an instance attribute

Igor provided a very nice piece of code. You will probably want to enhance the method factory to support non-binary operations but, apart from that, in my opinion the use of a blacklist is not ideal in term of maintenance. You have to carefully review all possible special methods now, and check them again for possibly new ones with each new release of python.

Building on top of Igor's code, I suggest another way making use of multiple inheritance. Inheriting from both the wrapped type and the value container lets the containers be almost perfectly compatible with the wrapped types while including the common services from the generic container. As a bonus, this approach makes the code even simpler (and even better with Igor's tip about lru_cache).

import functools

# container type superclass
class ValueDecorator:
    def wrapped_type(self):
        return type(self).__bases__[1]

    def custom_operation(self):
        print('hey! i am a', self.wrapped_type(), 'and a', type(self))

    def __repr__(self):
        return '{}({})'.format(self.__class__.__name__, super().__repr__())

# create and cache container types (e.g. IntContainer, StrContainer)
@functools.lru_cache(maxsize=16)
def type_container(type_):
    name = f'{type_.__name__.title()}Container'
    bases = (ValueDecorator, type_)
    return type(name, bases, {})

# create or lookup an appropriate container
def value_container(value):
    cls = type_container(type(value))
    return cls(value)

Please note that unlike Sam and Igor's methods, which reference the input object in the container, this method creates a new subclassed object initialized with the input object. It is fine for basic values but may cause undesirable effects for other types, depending on how their constructor deals with copy.

How to automatically add numeric special methods to a class?

So if I understood correctly, You want to inherit from a type and also want to intercept it's methods. Am I right?

You could do the same thing with metaprogramming and metaclasses. But I think using 'class decorator' is much more simpler. Here is my approach:

from functools import wraps

def intercept_numeric_methods(cls):
    def method_decorator(fn):

        @wraps(fn)
        def inner(*args, **kwargs):
            print('Intercepted')
            return fn(*args, **kwargs)

        return inner

    numeric_methods = {'__add__', '__mul__'}

    for method_name, method in int.__dict__.items():
        if method_name in numeric_methods:
            setattr(cls, method_name, method_decorator(method))

    return cls

@intercept_numeric_methods
class Myclass(int):
    pass

obj1 = Myclass(10)
obj2 = Myclass(20)

print(obj1 + obj2)

I've just added __add__ and __mul__. Add the name of the methods you want into numeric_methods set.

---

Related Topics