Explain _Dict_ Attribute

Why is the __dict__ attribute of this function an empty dictionary?

The __dict__ attribute of a function object stores attributes assigned to the function object.

>>> def foo():
...     a = 2
...     return a
...
>>> foo.bar = 12
>>> foo.__dict__
{'bar': 12}

Attributes of the function object are not related to the local variables that exist during the function call. The former is unique (there is one __dict__ per function object) the latter are not (a function may be called multiple times with separate local variables).

>>> def nfoo(n: int):
...     print(f'level {n} before:', locals())
...     if n > 0:
...        nfoo(n - 1)
...     print(f'level {n} after: ', locals())
...
>>> nfoo(2)
level 2 before: {'n': 2}
level 1 before: {'n': 1}
level 0 before: {'n': 0}
level 0 after:  {'n': 0}
level 1 after:  {'n': 1}
level 2 after:  {'n': 2}

Note how the locals of each level exist at the same time but hold separate values for the same name.

__dict__ Attribute of Ultimate Base Class, object in Python

The first part of your question is already answered by the linked answer: the __dict__ of instances is stored as a descriptor on the class. This is the A.__dict__['__dict__']. A.__dict__ on the other hand stores all the attributes of the A class - which itself is an instance of type. So actually it's type.__dict__['__dict__'] that provides these variables:

>>> type.__dict__['__dict__']
<attribute '__dict__' of 'type' objects>
>>> A.__dict__ == type.__dict__['__dict__'].__get__(A)
True

The reason why you're not seeing a __dict__ attribute on object is because it doesn't have one. This means you can't set instance variables on object instances:

>>> o = object()
>>> o.x = 1

AttributeError: 'object' object has no attribute 'x'

Similar behavior can be achieved for custom classes by defining __slots__:

>>> class B:
...     __slots__ = ()
... 
>>> vars(B)
mappingproxy({'__module__': '__main__', '__slots__': (), '__doc__': None})
>>> 
>>> b = B()
>>> b.x = 1

AttributeError: 'B' object has no attribute 'x'

What is the __dict__.__dict__ attribute of a Python class?

First of all A.__dict__.__dict__ is different from A.__dict__['__dict__']. The former doesn't exist and the latter is the __dict__ attribute that the instances of the class would have. It's a data descriptor object that returns the internal dictionary of attributes for the specific instance. In short, the __dict__ attribute of an object can't be stored in object's __dict__, so it's accessed through a descriptor defined in the class.

To understand this, you'd have to read the documentation of the descriptor protocol.

The short version:

1. For an instance a of a class A, access to a.__dict__ is provided by A.__dict__['__dict__'] which is the same as vars(A)['__dict__'].
2. For a class A, access to A.__dict__ is provided by type.__dict__['__dict__'] (in theory) which is the same as vars(type)['__dict__'].

---

The long version:

Both classes and objects provide access to attributes both through the attribute operator (implemented via the class or metaclass's __getattribute__), and the __dict__ attribute/protocol which is used by vars(ob).

For normal objects, the __dict__ object creates a separate dict object, which stores the attributes, and __getattribute__ first tries to access it and get the attributes from there (before attempting to look for the attribute in the class by utilizing the descriptor protocol, and before calling __getattr__). The __dict__ descriptor on the class implements the access to this dictionary.

• a.name is equivalent to trying those in order: type(a).__dict__['name'].__get__(a, type(a)) (only if type(a).__dict__['name'] is a data descriptor), a.__dict__['name'], type(a).__dict__['name'].__get__(a, type(a)), type(a).__dict__['name'].
• a.__dict__ does the same but skips the second step for obvious reasons.

As it's impossible for the __dict__ of an instance to be stored in itself, it's accessed through the descriptor protocol directly instead and is stored in a special field in the instance.

A similar scenario is true for classes, although their __dict__ is a special proxy object that pretends to be a dictionary (but might not be internally), and doesn't allow you to change it or replace it with another one. This proxy allows you, among all else, to access the attributes of a class that are specific to it, and not defined in one of its bases.

By default, a vars(cls) of an empty class carries three descriptors: __dict__ for storing the attributes of the instances, __weakref__ which is used internally by weakref, and __doc__ the docstring of the class. The first two might be gone if you define __slots__. Then you wouldn't have __dict__ and __weakref__ attributes, but instead you'd have a single class attribute for each slot. The attributes of the instance then wouldn't be stored in a dictionary, and access to them will be provided by the respective descriptors in the class.

---

And lastly, the inconsistency that A.__dict__ is different from A.__dict__['__dict__'] is because the attribute __dict__ is, by exception, never looked up in vars(A), so what is true for it isn't true for practically any other attribute you'd use. For example, A.__weakref__ is the same thing as A.__dict__['__weakref__']. If this inconsistency didn't exist, using A.__dict__ would not work, and you'd have to always use vars(A) instead.

Python __dict__

The attribute __dict__ is supposed to contain user defined attributes.

No, the __dict__ contains the dynamic attributes of an object. Those are not the only attributes an object can have however, the type of the object is usually also consulted to find attributes.

For example, the methods on a class can be found as attributes on an instance too. Many such attributes are descriptor objects and are bound to the object when looked up. This is the job of the __getattribute__ method all classes inherit from object; attributes on an object are resolved via type(object).__getattribute__(attribute_name), at which point the descriptors on the type as well as attributes directly set on the object (in the __dict__ mapping) are considered.

The __bases__ attribute of a class is provided by the class metatype, which is type() by default; it is a descriptor:

>>> class Foo:
...     pass
...
>>> Foo.__bases__
(<class 'object'>,)
>>> type.__dict__['__bases__']
<attribute '__bases__' of 'type' objects>
>>> type.__dict__['__bases__'].__get__(Foo, type)
(<class 'object'>,)

__dict__ just happens to be a place to store attributes that can have any valid string name. For classes that includes several standard attributes set when the class is created (__module__ and __doc__), and others that are there as descriptors for instances of a class (__dict__ and __weakref__). The latter must be added to the class, because a class itself also has those attributes, taken from type, again as descriptors.

So why is __bases__ a descriptor, but __doc__ is not? You can't set __bases__ to just anything, so the descriptor setter checks for specific conditions and is an opportunity to rebuild internal caches. The Python core developers use descriptors to restrict what can be set, or when setting a value requires additional work (like validation and updating internal structures).

Why are the class __dict__ and __weakref__ never re-defined in Python?

The '__dict__' and '__weakref__' entries in a class's __dict__ (when present) are descriptors used for retrieving an instance's dict pointer and weakref pointer from the instance memory layout. They're not the actual class's __dict__ and __weakref__ attributes - those are managed by descriptors on the metaclass.

There's no point adding those descriptors if a class's ancestors already provide one. However, a class does need its own __module__ and __doc__, regardless of whether its parents already have one - it doesn't make sense for a class to inherit its parent's module name or docstring.

---

You can see the implementation in type_new, the (very long) C implementation of type.__new__. Look for the add_weak and add_dict variables - those are the variables that determine whether type.__new__ should add space for __dict__ and __weakref__ in the class's instance memory layout. If type.__new__ decides it should add space for one of those attributes to the instance memory layout, it also adds getset descriptors to the class (through tp_getset) to retrieve the attributes:

if (add_dict) {
    if (base->tp_itemsize)
        type->tp_dictoffset = -(long)sizeof(PyObject *);
    else
        type->tp_dictoffset = slotoffset;
    slotoffset += sizeof(PyObject *);
}
if (add_weak) {
    assert(!base->tp_itemsize);
    type->tp_weaklistoffset = slotoffset;
    slotoffset += sizeof(PyObject *);
}
type->tp_basicsize = slotoffset;
type->tp_itemsize = base->tp_itemsize;
type->tp_members = PyHeapType_GET_MEMBERS(et);

if (type->tp_weaklistoffset && type->tp_dictoffset)
    type->tp_getset = subtype_getsets_full;
else if (type->tp_weaklistoffset && !type->tp_dictoffset)
    type->tp_getset = subtype_getsets_weakref_only;
else if (!type->tp_weaklistoffset && type->tp_dictoffset)
    type->tp_getset = subtype_getsets_dict_only;
else
    type->tp_getset = NULL;

If add_dict or add_weak are false, no space is reserved and no descriptor is added. One condition for add_dict or add_weak to be false is if one of the parents already reserved space:

add_dict = 0;
add_weak = 0;
may_add_dict = base->tp_dictoffset == 0;
may_add_weak = base->tp_weaklistoffset == 0 && base->tp_itemsize == 0;

This check doesn't actually care about any ancestor descriptors, just whether an ancestor reserved space for an instance dict pointer or weakref pointer, so if a C ancestor reserved space without providing a descriptor, the child won't reserve space or provide a descriptor. For example, set has a nonzero tp_weaklistoffset, but no __weakref__ descriptor, so descendants of set won't provide a __weakref__ descriptor either, even though instances of set (including subclass instances) support weak references.

You'll also see an && base->tp_itemsize == 0 in the initialization for may_add_weak - you can't add weakref support to a subclass of a class with variable-length instances.

How to get dictionary of sub-object using __dict__ on main object?

You can do something like this:

# Equivalent to your Engine class
class Foo:
    def __init__(self, value):
        self.v1 = value
        
# Equivalent to your Battery class
class Bar:
    def __init__(self, value):
        self.v2 = value
        
# Equivalent to your Wheels class
class Baz:
    def __init__(self, value):
        self.v3 = value
    
# Equivalent to your Car class
class FBB:
    def __init__(self):
        self.v = 0
        self.foo = Foo(1)
        self.bar = Bar(2)
        self.baz = Baz(3)
        
fbb = FBB()
print({k: vars(v) if hasattr(v, '__dict__') else v for k, v in vars(fbb).items()})
# {'v': 0, 'foo': {'v1': 1}, 'bar': {'v2': 2}, 'baz': {'v3': 3}}

---

Note that this won't work if Bar contains a reference to Baz, for example. For that, you may consider creating a recursive function:

class Foo:
    def __init__(self, value):
        self.v1 = value
        
class Bar:
    def __init__(self, value):
        self.v2 = value
        # Contains a reference to Baz!
        self.baz = Baz(3)
        
class Baz:
    def __init__(self, value):
        self.v3 = value
    
class FBB:
    def __init__(self):
        self.v = 0
        self.foo = Foo(1)
        self.bar = Bar(2)
        
def full_vars(obj):
    return {k: full_vars(v) if hasattr(v, '__dict__') else v for k, v in vars(obj).items()}
        
fbb = FBB()
print(full_vars(fbb))
# {'v': 0, 'foo': {'v1': 1}, 'bar': {'v2': 2, 'baz': {'v3': 3}}}

If an object doesn't have `__dict__`, must its class have a `__slots__` attribute?

For user defined classes (defined using the class keyword in regular Python code), a class will always have __slots__ on the class, __dict__ on the instance, or both (if one of the slots defined is '__dict__', or one of the user defined classes in an inheritance chain defines __slots__ and another one does not, creating __dict__ implicitly). So that's three of four possibilities covered for user defined classes.

Edit: A correction: Technically, a user-defined class could have neither; the class would be defined with __slots__, but have it deleted after definition time (the machinery that sets up the type doesn't require __slots__ to persist after the class definition finishes). No sane person should do this, and it could have undesirable side-effects (full behavior untested), but it's possible.

For built-in types, at least in the CPython reference interpreter, they're extremely unlikely to have __slots__ (if they did, it would be to simulate a user-defined class, defining it doesn't actually do anything useful). A built-in type typically stores its attributes as raw C level values and pointers on a C level struct, optionally with explicitly created descriptors or accessor methods, which eliminates the purpose of __slots__, which are just a convenient limited purpose equivalent of such struct games for user defined classes. __dict__ is opt-in for built-in types, not on by default (though the opt-in process is fairly easy; you need to put a PyObject* entry somewhere in the struct and provide the offset to it in the type definition).

To be clear, __dict__ need not appear on the class for it to appear on its instances; __slots__ is class level, and can suppress the __dict__ on the instance, but has no effect on whether the class itself has a __dict__; user defined classes always have __dict__, but their instances won't if you're careful to use __slots__ appropriately.

So in short:

(Sane) User defined classes have at least one of __dict__ (on the instances) or __slots__ (on the class), and can have both. Insane user defined classes could have neither, but only a deranged developer would do it.

Built-in classes often have neither, may provide __dict__, and almost never provide __slots__ as it is pointless for them.

Examples:

# Class has __slots__, instances don't have __dict__
class DictLess:
    __slots__ = ()

# Instances have __dict__, class lacks __slots__
class DictOnly:
    pass

# Class has __slots__, instances have __dict__ because __slots__ declares it
class SlottedDict:
    __slots__ = '__dict__',

# Class has __slots__ without __dict__ slot, instances have it anyway from unslotted parent
class DictFromParent(DictOnly):
    __slots__ = ()

# Complete insanity: __slots__ takes effect at class definition time, but can
# be deleted later, without changing the class behavior:
class NoSlotNoDict:
    __slots__ = ()
del NoSlotNoDict.__slots__
# Instances have no __dict__, class has no __slots__ but acts like it does
# (the machinery to make it slotted isn't undone by deleting __slots__)
# Please, please don't actually do this

# Built-in type without instance __dict__ or class defined __slots__:
int().__dict__  # Raises AttributeError
int.__slots__   # Also raises AttributeError

# Built-in type that opts in to __dict__ on instances:
import functools
functools.partial(int).__dict__ # Works fine
functools.partial.__slots__ # Raises AttributeError

When does Python fall back onto class __dict__ from instance __dict__?

According to (already mentioned) [Python.Docs]: Data model (emphasis is mine):

Custom classes

Custom class types are typically created by class definitions (see section Class definitions). A class has a namespace implemented by a dictionary object. Class attribute references are translated to lookups in this dictionary, e.g., C.x is translated to C.__dict__["x"] (although there are a number of hooks which allow for other means of locating attributes). When the attribute name is not found there, the attribute search continues in the base classes.

...

Class instances

A class instance is created by calling a class object (see above). A class instance has a namespace implemented as a dictionary which is the first place in which attribute references are searched. When an attribute is not found there, and the instance’s class has an attribute by that name, the search continues with the class attributes.

...

Invoking Descriptors

...

The default behavior for attribute access is to get, set, or delete the attribute from an object’s dictionary. For instance, a.x has a lookup chain starting with a.__dict__['x'], then type(a).__dict__['x'], and continuing through the base classes of type(a) excluding metaclasses.

However, if the looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead. Where this occurs in the precedence chain depends on which descriptor methods were defined and how they were called.

Attributes defined inside a class definition (but outside the initializer (or other methods)) are called class attributes, and are bound to the class itself rather than its instances. It's like static members from C++ or Java. [Python.Docs]: Compound statements - Class definitions states (emphasis still mine):

Programmer’s note: Variables defined in the class definition are class attributes; they are shared by instances. Instance attributes can be set in a method with self.name = value. Both class and instance attributes are accessible through the notation “self.name”, and an instance attribute hides a class attribute with the same name when accessed in this way. Class attributes can be used as defaults for instance attributes, but using mutable values there can lead to unexpected results. Descriptors can be used to create instance variables with different implementation details.

So, the attribute lookup order can be summarized like below (traverse in ascending order, when attribute name found simply return its value (therefore ignoring the remaining entries)). The first steps performed by the (builtin) __getattribute__ method:

0. Descriptors (if any - note that their presence could also be triggered indirectly (by other features))

1. Instance namespace (foo.__dict__)

2. Instance class namespace (Foo.__dict__)

3. Instance class base classes namespaces (e.__dict__ for e in Foo.__mro__)

4. Anything that a custom __getattr__ method might return

The above is what typically happens, as Python being highly customizable that can be altered (e.g. __slots__).

For an exact behavior, you could check the source code ([GitHub]: python/cpython - (main) cpython/Objects):

• typeobject.c: type_getattro (optionally: super_getattro, slot_tp_getattro)

• object.c: _PyObject_GenericGetAttrWithDict

Here's an example that will clear things up (hopefully).

code00.py:

#!/usr/bin/env python

import sys
from pprint import pformat as pf

def print_dict(obj, header="", indent=0, filterfunc=lambda x, y: not x.startswith("__")):
    if not header:
        header = getattr(obj, "__name__", None)
    if header:
        print("{:}{:}.__dict__:".format("  " * indent, header))
    lines = pf({k: v for k, v in getattr(obj, "__dict__", {}).items() if filterfunc(k, v)}, sort_dicts=False).split("\n")
    for line in lines:
        print("{:}{:}".format("  " * (indent + 1), line))
    print()

class Descriptor:
    def __init__(self, name):
        self.name = name

    def __get__(self, instance, cls):
        print("{:s}.__get__".format(self.name))

    def __set__(self, instance, value):
        print("{:s}.__set__ - {:}".format(self.name, value))

    def __delete__(self, instance):
        print("{:s}.__delete__".format(self.name))

class Demo:
    cls_attr0 = 3.141593
    cls_attr1 = Descriptor("cls_attr1")

    '''
    def __getattribute__(self, name):
        print("__getattribute__:", self, name)
        return super().__getattribute__(name)
    '''

    '''
    def __getattr__(self, name):
        print("__getattr__:", self, name)
        return "something dummy"
    '''

    def __init__(self):
        self.inst_attr0 = 2.718282

def main(*argv):
    print("ORIGINAL")
    demos = [Demo() for _ in range(2)]
    demo0 = demos[0]
    demo1 = demos[1]
    print_dict(Demo)
    print_dict(demo0, header="demo0")
    print("\ndemo0 attrs:", demo0.cls_attr0, demo0.cls_attr1, demo0.inst_attr0)
    print_dict(demo1, header="\ndemo1")
    print("\ndemo1 attrs:", demo1.cls_attr0, demo1.cls_attr1, demo1.inst_attr0)

    print("\nALTER 1ST INSTANCE OBJECT")
    demo0.inst_attr0 = -3
    demo0.cls_attr0 = -5

    print_dict(Demo)
    print_dict(demo0, header="demo0")
    print("\ndemo0 attrs:", demo0.cls_attr0, demo0.cls_attr1, demo0.inst_attr0)
    print_dict(demo1, header="\ndemo1")
    print("\ndemo1 attrs:", demo1.cls_attr0, demo1.cls_attr1, demo1.inst_attr0)

    print("\nALTER CLASS")
    Demo.cls_attr0 = -7
    Demo.cls_attr1 = -9
    print_dict(Demo, header="Demo")
    print_dict(demo1, header="demo0")
    print("\ndemo0 attrs:", demo0.cls_attr0, demo0.cls_attr1, demo0.inst_attr0)
    print_dict(demo1, header="\ndemo1")
    print("\ndemo1 attrs:", demo1.cls_attr0, demo1.cls_attr1, demo1.inst_attr0)

if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q072399556]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" code00.py
Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32

ORIGINAL
Demo.__dict__:
  {'cls_attr0': 3.141593,
   'cls_attr1': <__main__.Descriptor object at 0x00000171B0B24FD0>}

demo0.__dict__:
  {'inst_attr0': 2.718282}

cls_attr1.__get__

demo0 attrs: 3.141593 None 2.718282

demo1.__dict__:
  {'inst_attr0': 2.718282}

cls_attr1.__get__

demo1 attrs: 3.141593 None 2.718282

ALTER 1ST INSTANCE OBJECT
Demo.__dict__:
  {'cls_attr0': 3.141593,
   'cls_attr1': <__main__.Descriptor object at 0x00000171B0B24FD0>}

demo0.__dict__:
  {'inst_attr0': -3, 'cls_attr0': -5}

cls_attr1.__get__

demo0 attrs: -5 None -3

demo1.__dict__:
  {'inst_attr0': 2.718282}

cls_attr1.__get__

demo1 attrs: 3.141593 None 2.718282

ALTER CLASS
Demo.__dict__:
  {'cls_attr0': -7, 'cls_attr1': -9}

demo0.__dict__:
  {'inst_attr0': 2.718282}

demo0 attrs: -5 -9 -3

demo1.__dict__:
  {'inst_attr0': 2.718282}

demo1 attrs: -7 -9 2.718282

Done.

---

Related Topics