SchröDinger's Variable: the _Class_ Cell Magically Appears If You'Re Checking for Its Presence

Schrödinger's variable: the __class__ cell magically appears if you're checking for its presence?

This is a weird interaction in Python 3's implementation of no-argument super. An access to super in a method triggers the addition of a hidden __class__ closure variable referring to the class that defines the method. The parser special-cases a load of the name super in a method by also adding __class__ to the method's symbol table, and then the rest of the relevant code all looks for __class__ instead of super. However, if you try to access __class__ yourself, all the code looking for __class__ sees it and thinks it should do the super handling!

Here's where it adds the name __class__ to the symbol table if it sees super:

case Name_kind:
    if (!symtable_add_def(st, e->v.Name.id,
                          e->v.Name.ctx == Load ? USE : DEF_LOCAL))
        VISIT_QUIT(st, 0);
    /* Special-case super: it counts as a use of __class__ */
    if (e->v.Name.ctx == Load &&
        st->st_cur->ste_type == FunctionBlock &&
        !PyUnicode_CompareWithASCIIString(e->v.Name.id, "super")) {
        if (!GET_IDENTIFIER(__class__) ||
            !symtable_add_def(st, __class__, USE))
            VISIT_QUIT(st, 0);
    }
    break;

Here's drop_class_free, which sets ste_needs_class_closure:

static int
drop_class_free(PySTEntryObject *ste, PyObject *free)
{
    int res;
    if (!GET_IDENTIFIER(__class__))
        return 0;
    res = PySet_Discard(free, __class__);
    if (res < 0)
        return 0;
    if (res)
        ste->ste_needs_class_closure = 1;
    return 1;
}

The compiler section that checks ste_needs_class_closure and creates the implicit cell:

if (u->u_ste->ste_needs_class_closure) {
    /* Cook up an implicit __class__ cell. */
    _Py_IDENTIFIER(__class__);
    PyObject *tuple, *name, *zero;
    int res;
    assert(u->u_scope_type == COMPILER_SCOPE_CLASS);
    assert(PyDict_Size(u->u_cellvars) == 0);
    name = _PyUnicode_FromId(&PyId___class__);
    if (!name) {
        compiler_unit_free(u);
        return 0;
    }
    ...

There's more relevant code, but it's too much to include all of it. Python/compile.c and Python/symtable.c are where to look if you want to see more.

You can get some weird bugs if you try to use a variable named __class__:

class Foo:
    def f(self):
        __class__ = 3
        super()

Foo().f()

Output:

Traceback (most recent call last):
  File "./prog.py", line 6, in <module>
  File "./prog.py", line 4, in f
RuntimeError: super(): __class__ cell not found

The assignment to __class__ means __class__ is a local variable instead of a closure variable, so the closure cell super() needs isn't there.

def f():
    __class__ = 2
    class Foo:
        def f(self):
            print(__class__)

    Foo().f()

f()

Output:

<class '__main__.f.<locals>.Foo'>

Even though there's an actual __class__ variable in the enclosing scope, the special-casing of __class__ means you get the class instead of the enclosing scope's variable value.

Why does assigning to the __class__ cell break `super`?

You need a nonlocal statement to assign to closure variables, including the magic __class__ closure variable. Assigning to __class__ without a nonlocal statement creates a local variable that hides the magic closure variable.

You're expecting __class__ to behave as if it was local to meth, but it actually behaves as if it's local to an invisible pseudo-scope in which all methods of Demo are nested. If it was treated as local to meth, you wouldn't need nonlocal.

If you do add a nonlocal statement, the implementation actually will allow you to reassign the magic closure variable:

class Foo:
    def meth(self):
        nonlocal __class__
        __class__ = 3
        super()

Foo().meth()

Result:

Traceback (most recent call last):
  File "./prog.py", line 7, in <module>
  File "./prog.py", line 5, in meth
RuntimeError: super(): __class__ is not a type (int)

Why __class__ can be directly accessed

So, it is a quirk for the compiler to find the parent class.

From Python Data Model:

__class__ is an implicit closure reference created by the compiler if any methods in a class body refer to either __class__ or super. This
allows the zero argument form of super() to correctly identify the
class being defined based on lexical scoping, while the class or
instance that was used to make the current call is identified based on
the first argument passed to the method.

How to get the class from which a method was called?

Walking the stack should give the answer.

The answer should ideally be, in the caller's stack frame.

The problem is, the stack frames only record the function

names (like so: 'f', 'g', 'h', etc.) Any information about

classes is lost. Trying to reverse-engineer the lost info,

by navigating the class hierarchy (in parallel with the

stack frame), did not get me very far, and got complicated.

So, here is a different approach:

Inject the class info into the stack frame

(e.g. with local variables),

and read that, from the called function.

import inspect

class A:
  def f(self):
    frame = inspect.currentframe()
    callerFrame = frame.f_back
    callerLocals = callerFrame.f_locals
    return callerLocals['cls']

class B(A):
  def g(self):
    cls = B
    return self.f()
    
  def f(self):
    cls = B
    return super().f()

class C(B):
  def h(self):
    cls = C
    return super(B, self).f()
  
  def f(self):
    cls = C
    return super().f()

c = C()
assert c.h() == C
assert c.g() == B
assert c.f() == B

Related:

get-fully-qualified-method-name-from-inspect-stack

---

Without modifying the definition of subclasses:

Added an "external" decorator, to wrap class methods.

(At least as a temporary solution.)

import inspect

class Injector:
  def __init__(self, nameStr, valueStr):
    self.nameStr = nameStr
    self.valueStr = valueStr
  
  # Should inject directly in f's local scope / stack frame.
  # As is, it just adds another stack frame on top of f.
  def injectInLocals(self, f):
    def decorate(*args, **kwargs):
      exec(f'{self.nameStr} = {self.valueStr}')
      return f(*args, **kwargs)
    return decorate

class A:
  def f(self):
    frame = inspect.currentframe()
    callerDecoratorFrame = frame.f_back.f_back  # Note:twice
    callerDecoratorLocals = callerDecoratorFrame.f_locals
    return callerDecoratorLocals['cls']

class B(A):
  def g(self): return self.f()
  def f(self): return super().f()

class C(B):
  def h(self): return super(B, self).f()
  def f(self): return super().f()

bInjector = Injector('cls', B.__name__)
B.g = bInjector.injectInLocals(B.g)
B.f = bInjector.injectInLocals(B.f)

cInjector = Injector('cls', C.__name__)
C.h = cInjector.injectInLocals(C.h)
C.f = cInjector.injectInLocals(C.f)

c = C()
assert c.h() == C
assert c.g() == B
assert c.f() == B

I found this link very interesting

(but didn't take advantage of metaclasses here):

what-are-metaclasses-in-python

Maybe someone could even replace the function definitions*,

with functions whose code is a duplicate of the original;

but with added locals/information, directly in their scope.

*

Maybe after the class definitions have completed;

maybe during class creation (using a metaclass).

Why does super closure not use the new base class given by a metaclass?

The problem here is that your dynamic class does not inherit from itself (Class) at all - the implicit __class__ variable inside your __init__ points to the "hardcoded" "Class" version - but the self received when __init__ is called is an instance of the dynamic class, which does not have Class as its superclass. Thus, the arguments filled implicitly to super(): __class__ and self will mismatch (self is not an instance of the class defined in __class__ or of a subclass of it).

The reliable way to fix this is to allow proper inheritance, and forget copying the class __dict__ attributes around: let the inheritance machinery take care of calling the methods in the appropriate order.

By simply making the dynamic class inherit from your static class and the dynamic-base, all methods are in place, and self will be a proper instance from the baked-in __class__ from __init__.__class__ still points to the static Class, but the conditions for super to be called are fulfilled - and super does the right thing, by looking for the supermethods starting from the self parameter - the newly created dynamic subclass which inherits both from your static Class and the new Base, and calls the methods on Base as they are part of the __mro__ of the new class, in the correct place.

Ok - sounds complicated - but with some print statements added we can see the thing working:

class Base:
    def __init__(self):
        print("at base __init__")

class Meta(type):
    def __call__(cls, obj, *args, **kwargs):
        dynamic_ancestor = type(obj)
        bases = (cls, dynamic_ancestor,)

        new_cls = type(f"{cls.__name__}__{dynamic_ancestor.__name__}", bases , {})
        instance = new_cls.__new__(new_cls, *args, **kwargs)
        instance.__init__(*args, **kwargs)
        return instance

class Class(metaclass=Meta):
    def __init__(self, *args, **kwargs):
        print(__class__.__bases__)
        print(self.__class__.__bases__)
        print(isinstance(self, Class))
        print(isinstance(Class, Meta))
        print(isinstance(self, __class__))
        print(isinstance(self, self.__class__))
        print(self.__class__.__mro__, __class__.__mro__)
        super().__init__(*args, **kwargs)

Class(Base())

Outputs:

at base __init__
(<class 'object'>,)
(<class '__main__.Class'>, <class '__main__.Base'>)
True
True
True
True
(<class '__main__.Class__Base'>, <class '__main__.Class'>, <class '__main__.Base'>, <class 'object'>) (<class '__main__.Class'>, <class 'object'>)
at base __init__

Why can't you omit the arguments to super if you add *args in __init__ definition?

Per PEP 3135, which introduced "new super" (emphasis mine):

The new syntax:

super()

is equivalent to:

super(__class__, <firstarg>)

where __class__ is the class that the method was defined in, and
<firstarg> is the first parameter of the method (normally self for
instance methods, and cls for class methods).

There must be a specific first parameter for this to work (although it doesn't necessarily have to be called self or cls), it won't use e.g. args[0].

---

As to why it needs to be a specific parameter, that's due to the implementation; per the comment it uses the "first local variable on the stack". If co->co_argcount == 0, as it is when you only specify *args, you get the no arguments error. This behaviour may not be the same in other implementations than CPython.

---

Related

• How is super() in Python 3 implemented?
• Why is Python 3.x's super() magic?
• Get "super(): no arguments" error in one case but not a similar case
• Schrödinger's variable: the __class__ cell magically appears if you're checking for its presence?

super() throwing an error in Sublime Text, works in PyCharm/Terminal

Your sublimetext is using the default build system, which is Python 2. Configure it to run in Python 3.

Tools -> Build System -> New Build System ...

Add this content:

{
    "cmd": ["python3", "-u", "$file"],
    "file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
    "selector": "source.python"
}

Save the configuration with a sensible filename, say python3.sublime-build, and select this newly created build in Tools -> Build With ....

---

Related Topics