What is the __dict__.__dict__ attribute of a Python class?
First of all A.__dict__.__dict__
is different from A.__dict__['__dict__']
. The former doesn't exist and the latter is the __dict__
attribute that the instances of the class would have. It's a data descriptor object that returns the internal dictionary of attributes for the specific instance. In short, the __dict__
attribute of an object can't be stored in object's __dict__
, so it's accessed through a descriptor defined in the class.
To understand this, you'd have to read the documentation of the descriptor protocol.
The short version:
- For an instance
a
of a classA
, access toa.__dict__
is provided byA.__dict__['__dict__']
which is the same asvars(A)['__dict__']
. - For a class
A
, access toA.__dict__
is provided bytype.__dict__['__dict__']
(in theory) which is the same asvars(type)['__dict__']
.
The long version:
Both classes and objects provide access to attributes both through the attribute operator (implemented via the class or metaclass's __getattribute__
), and the __dict__
attribute/protocol which is used by vars(ob)
.
For normal objects, the __dict__
object creates a separate dict
object, which stores the attributes, and __getattribute__
first tries to access it and get the attributes from there (before attempting to look for the attribute in the class by utilizing the descriptor protocol, and before calling __getattr__
). The __dict__
descriptor on the class implements the access to this dictionary.
a.name
is equivalent to trying those in order:type(a).__dict__['name'].__get__(a, type(a))
(only iftype(a).__dict__['name']
is a data descriptor),a.__dict__['name']
,type(a).__dict__['name'].__get__(a, type(a))
,type(a).__dict__['name']
.a.__dict__
does the same but skips the second step for obvious reasons.
As it's impossible for the __dict__
of an instance to be stored in itself, it's accessed through the descriptor protocol directly instead and is stored in a special field in the instance.
A similar scenario is true for classes, although their __dict__
is a special proxy object that pretends to be a dictionary (but might not be internally), and doesn't allow you to change it or replace it with another one. This proxy allows you, among all else, to access the attributes of a class that are specific to it, and not defined in one of its bases.
By default, a vars(cls)
of an empty class carries three descriptors: __dict__
for storing the attributes of the instances, __weakref__
which is used internally by weakref
, and __doc__
the docstring of the class. The first two might be gone if you define __slots__
. Then you wouldn't have __dict__
and __weakref__
attributes, but instead you'd have a single class attribute for each slot. The attributes of the instance then wouldn't be stored in a dictionary, and access to them will be provided by the respective descriptors in the class.
And lastly, the inconsistency that A.__dict__
is different from A.__dict__['__dict__']
is because the attribute __dict__
is, by exception, never looked up in vars(A)
, so what is true for it isn't true for practically any other attribute you'd use. For example, A.__weakref__
is the same thing as A.__dict__['__weakref__']
. If this inconsistency didn't exist, using A.__dict__
would not work, and you'd have to always use vars(A)
instead.
__dict__ Attribute of Ultimate Base Class, object in Python
The first part of your question is already answered by the linked answer: the __dict__
of instances is stored as a descriptor on the class. This is the A.__dict__['__dict__']
. A.__dict__
on the other hand stores all the attributes of the A
class - which itself is an instance of type
. So actually it's type.__dict__['__dict__']
that provides these variables:
>>> type.__dict__['__dict__']
<attribute '__dict__' of 'type' objects>
>>> A.__dict__ == type.__dict__['__dict__'].__get__(A)
True
The reason why you're not seeing a __dict__
attribute on object
is because it doesn't have one. This means you can't set instance variables on object
instances:
>>> o = object()
>>> o.x = 1
AttributeError: 'object' object has no attribute 'x'
Similar behavior can be achieved for custom classes by defining __slots__
:
>>> class B:
... __slots__ = ()
...
>>> vars(B)
mappingproxy({'__module__': '__main__', '__slots__': (), '__doc__': None})
>>>
>>> b = B()
>>> b.x = 1
AttributeError: 'B' object has no attribute 'x'
Why are the class __dict__ and __weakref__ never re-defined in Python?
The '__dict__'
and '__weakref__'
entries in a class's __dict__
(when present) are descriptors used for retrieving an instance's dict pointer and weakref pointer from the instance memory layout. They're not the actual class's __dict__
and __weakref__
attributes - those are managed by descriptors on the metaclass.
There's no point adding those descriptors if a class's ancestors already provide one. However, a class does need its own __module__
and __doc__
, regardless of whether its parents already have one - it doesn't make sense for a class to inherit its parent's module name or docstring.
You can see the implementation in type_new
, the (very long) C implementation of type.__new__
. Look for the add_weak
and add_dict
variables - those are the variables that determine whether type.__new__
should add space for __dict__
and __weakref__
in the class's instance memory layout. If type.__new__
decides it should add space for one of those attributes to the instance memory layout, it also adds getset descriptors to the class (through tp_getset
) to retrieve the attributes:
if (add_dict) {
if (base->tp_itemsize)
type->tp_dictoffset = -(long)sizeof(PyObject *);
else
type->tp_dictoffset = slotoffset;
slotoffset += sizeof(PyObject *);
}
if (add_weak) {
assert(!base->tp_itemsize);
type->tp_weaklistoffset = slotoffset;
slotoffset += sizeof(PyObject *);
}
type->tp_basicsize = slotoffset;
type->tp_itemsize = base->tp_itemsize;
type->tp_members = PyHeapType_GET_MEMBERS(et);
if (type->tp_weaklistoffset && type->tp_dictoffset)
type->tp_getset = subtype_getsets_full;
else if (type->tp_weaklistoffset && !type->tp_dictoffset)
type->tp_getset = subtype_getsets_weakref_only;
else if (!type->tp_weaklistoffset && type->tp_dictoffset)
type->tp_getset = subtype_getsets_dict_only;
else
type->tp_getset = NULL;
If add_dict
or add_weak
are false, no space is reserved and no descriptor is added. One condition for add_dict
or add_weak
to be false is if one of the parents already reserved space:
add_dict = 0;
add_weak = 0;
may_add_dict = base->tp_dictoffset == 0;
may_add_weak = base->tp_weaklistoffset == 0 && base->tp_itemsize == 0;
This check doesn't actually care about any ancestor descriptors, just whether an ancestor reserved space for an instance dict pointer or weakref pointer, so if a C ancestor reserved space without providing a descriptor, the child won't reserve space or provide a descriptor. For example, set
has a nonzero tp_weaklistoffset
, but no __weakref__
descriptor, so descendants of set
won't provide a __weakref__
descriptor either, even though instances of set
(including subclass instances) support weak references.
You'll also see an && base->tp_itemsize == 0
in the initialization for may_add_weak
- you can't add weakref support to a subclass of a class with variable-length instances.
Where do classes get their default '__dict__' attributes from?
The
__dict__
attribute is created by the internal code intype.__new__
. The class's metaclass may influence the eventual content of__dict__
. If you are using__slots__
, you will have no__dict__
attribute.__module__
is set when the class is compiled, so the instance inherits this attribute from the class. You can verify this by performing the testinst.__module__ is cls.__module__
(given thatinst
is an instance ofcls
), and also by modifyingcls.__module__
and observing thatinst.__module__
reflects this change.(you can also do stupid things like setting
inst.__module__
to a differing value fromcls.__module__
. Not sure why this isn't a readonly attribute.)__weakref__
is created when the instance is; AFAIK it's completely controlled by the internal CPython object-creation system. This attribute is only present on instances ofobject
subclasses -- for example instances ofint
,list
,set
,tuple
have no__weakref__
attribute.In interpreting this explanation, keep in mind that
class C:
is equivalent toclass C(object):
for Python3. For Python2, plainclass C:
will generate instances without__weakref__
(or a number of other attributes acquired from theobject
implementation).
When does Python fall back onto class __dict__ from instance __dict__?
According to (already mentioned) [Python.Docs]: Data model (emphasis is mine):
Custom classes
Custom class types are typically created by class definitions (see section Class definitions). A class has a namespace implemented by a dictionary object. Class attribute references are translated to lookups in this dictionary, e.g.,
C.x
is translated toC.__dict__["x"]
(although there are a number of hooks which allow for other means of locating attributes). When the attribute name is not found there, the attribute search continues in the base classes....
Class instances
A class instance is created by calling a class object (see above). A class instance has a namespace implemented as a dictionary which is the first place in which attribute references are searched. When an attribute is not found there, and the instance’s class has an attribute by that name, the search continues with the class attributes.
...
Invoking Descriptors
...
The default behavior for attribute access is to get, set, or delete the attribute from an object’s dictionary. For instance,
a.x
has a lookup chain starting witha.__dict__['x']
, thentype(a).__dict__['x']
, and continuing through the base classes oftype(a)
excluding metaclasses.However, if the looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead. Where this occurs in the precedence chain depends on which descriptor methods were defined and how they were called.
Attributes defined inside a class definition (but outside the initializer (or other methods)) are called class attributes, and are bound to the class itself rather than its instances. It's like static members from C++ or Java. [Python.Docs]: Compound statements - Class definitions states (emphasis still mine):
Programmer’s note: Variables defined in the class definition are class attributes; they are shared by instances. Instance attributes can be set in a method with
self.name = value
. Both class and instance attributes are accessible through the notation “self.name
”, and an instance attribute hides a class attribute with the same name when accessed in this way. Class attributes can be used as defaults for instance attributes, but using mutable values there can lead to unexpected results. Descriptors can be used to create instance variables with different implementation details.
So, the attribute lookup order can be summarized like below (traverse in ascending order, when attribute name found simply return its value (therefore ignoring the remaining entries)). The first steps performed by the (builtin) __getattribute__ method:
Descriptors (if any - note that their presence could also be triggered indirectly (by other features))
Instance namespace (foo.__dict__)
Instance class namespace (Foo.__dict__)
Instance class base classes namespaces (
e.__dict__ for e in Foo.__mro__
)Anything that a custom __getattr__ method might return
The above is what typically happens, as Python being highly customizable that can be altered (e.g. __slots__).
For an exact behavior, you could check the source code ([GitHub]: python/cpython - (main) cpython/Objects):
typeobject.c: type_getattro (optionally: super_getattro, slot_tp_getattro)
object.c: _PyObject_GenericGetAttrWithDict
Here's an example that will clear things up (hopefully).
code00.py:
#!/usr/bin/env python
import sys
from pprint import pformat as pf
def print_dict(obj, header="", indent=0, filterfunc=lambda x, y: not x.startswith("__")):
if not header:
header = getattr(obj, "__name__", None)
if header:
print("{:}{:}.__dict__:".format(" " * indent, header))
lines = pf({k: v for k, v in getattr(obj, "__dict__", {}).items() if filterfunc(k, v)}, sort_dicts=False).split("\n")
for line in lines:
print("{:}{:}".format(" " * (indent + 1), line))
print()
class Descriptor:
def __init__(self, name):
self.name = name
def __get__(self, instance, cls):
print("{:s}.__get__".format(self.name))
def __set__(self, instance, value):
print("{:s}.__set__ - {:}".format(self.name, value))
def __delete__(self, instance):
print("{:s}.__delete__".format(self.name))
class Demo:
cls_attr0 = 3.141593
cls_attr1 = Descriptor("cls_attr1")
'''
def __getattribute__(self, name):
print("__getattribute__:", self, name)
return super().__getattribute__(name)
'''
'''
def __getattr__(self, name):
print("__getattr__:", self, name)
return "something dummy"
'''
def __init__(self):
self.inst_attr0 = 2.718282
def main(*argv):
print("ORIGINAL")
demos = [Demo() for _ in range(2)]
demo0 = demos[0]
demo1 = demos[1]
print_dict(Demo)
print_dict(demo0, header="demo0")
print("\ndemo0 attrs:", demo0.cls_attr0, demo0.cls_attr1, demo0.inst_attr0)
print_dict(demo1, header="\ndemo1")
print("\ndemo1 attrs:", demo1.cls_attr0, demo1.cls_attr1, demo1.inst_attr0)
print("\nALTER 1ST INSTANCE OBJECT")
demo0.inst_attr0 = -3
demo0.cls_attr0 = -5
print_dict(Demo)
print_dict(demo0, header="demo0")
print("\ndemo0 attrs:", demo0.cls_attr0, demo0.cls_attr1, demo0.inst_attr0)
print_dict(demo1, header="\ndemo1")
print("\ndemo1 attrs:", demo1.cls_attr0, demo1.cls_attr1, demo1.inst_attr0)
print("\nALTER CLASS")
Demo.cls_attr0 = -7
Demo.cls_attr1 = -9
print_dict(Demo, header="Demo")
print_dict(demo1, header="demo0")
print("\ndemo0 attrs:", demo0.cls_attr0, demo0.cls_attr1, demo0.inst_attr0)
print_dict(demo1, header="\ndemo1")
print("\ndemo1 attrs:", demo1.cls_attr0, demo1.cls_attr1, demo1.inst_attr0)
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.")
sys.exit(rc)
Output:
[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q072399556]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" code00.py
Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32
ORIGINAL
Demo.__dict__:
{'cls_attr0': 3.141593,
'cls_attr1': <__main__.Descriptor object at 0x00000171B0B24FD0>}
demo0.__dict__:
{'inst_attr0': 2.718282}
cls_attr1.__get__
demo0 attrs: 3.141593 None 2.718282
demo1.__dict__:
{'inst_attr0': 2.718282}
cls_attr1.__get__
demo1 attrs: 3.141593 None 2.718282
ALTER 1ST INSTANCE OBJECT
Demo.__dict__:
{'cls_attr0': 3.141593,
'cls_attr1': <__main__.Descriptor object at 0x00000171B0B24FD0>}
demo0.__dict__:
{'inst_attr0': -3, 'cls_attr0': -5}
cls_attr1.__get__
demo0 attrs: -5 None -3
demo1.__dict__:
{'inst_attr0': 2.718282}
cls_attr1.__get__
demo1 attrs: 3.141593 None 2.718282
ALTER CLASS
Demo.__dict__:
{'cls_attr0': -7, 'cls_attr1': -9}
demo0.__dict__:
{'inst_attr0': 2.718282}
demo0 attrs: -5 -9 -3
demo1.__dict__:
{'inst_attr0': 2.718282}
demo1 attrs: -7 -9 2.718282
Done.
If one of the attribute of my class is a dictionary, how to define the set property for it?
What happens in your code here myobject.params_dict["p2"] = 10
is:
You access a property params_dict
which returned _params_dict
of the instance... then you assign 10
to its item p2
. The setter never comes to play and also could not... not like this. If you wanted to use a property setter, you could:
class MyClass:
...
def _set_params_dict(self, params_dict):
self._params_dict.update(params_dict)
...
myobject.params_dict = {"p2": 10}
Setter which updates the dict is used, when you assign to the property.
This should explain what went down and why. How to do this "properly"... depends on what exactly are you after.
Side note: Using this:
def __init__(self, a, b, params_dict={"p1": 1, "p2":-0.136})
self._params_dict = params_dict
Means that when your class is created a dictionary object is created and assigned as default parameter for the constructor... which is then assigned to an attribute of the instance.
In turn this implies, all instances where default params_dict
is used (no value was supplied) share the same one dictionary object for _params_dict
.
To address your comment: it's not a matter of (my) opinion. You can get exactly the interface usable the way you've suggested having custom action performed when setting an item, but you would have to modify the _params_dict
type to do that, e.g.:
class ParamsDict(dict):
def __setitem__(self, key, value, /):
print(f"Special action setting {key}")
super().__setitem__(key, value)
class MyClass:
def __init__(self, a, b, params_dict=(("p1", 1), ("p2", -0.136))):
self.a = a
self.b = b
self.params_dict = ParamsDict(params_dict)
myobject = MyClass(a=0, b=4, params_dict={"p1": 1, "p2":-0.136, "p3":89})
myobject.params_dict["p2"] = 20
You could also consider just exposing items of params_dict
attribute as items of MyClass
if that makes sense for your use case:
class MyClass:
def __init__(self, a, b, params_dict=None):
if params_dict is None:
param_dict = {"p1": 1, "p2": -0.136}
self.a = a
self.b = b
self.params_dict = params_dict
def __getitem__(self, key):
return self.params_dict[key]
def __setitem__(self, key, value):
print(f"Special action setting {key}")
self.params_dict[key] = value
myobject = MyClass(a=0, b=4, params_dict={"p1": 1, "p2":-0.136, "p3":89})
myobject["p2"] = 20
print(myobject["p2"])
But without much more context and understanding why (where is all this heading) by far the easiest (and likely cleanest) solution would often be to not bother with either of the above and just define a method to do do this - set a parameter value and perform whatever other operations you need, e.g.:
class MyClass:
def __init__(self, a, b, params_dict=None):
if params_dict is None:
param_dict = {"p1": 1, "p2": -0.136}
self.a = a
self.b = b
self.params_dict = params_dict
def set_param(self, key, value):
print(f"Special action setting {key}")
self.params_dict[key] = value
myobject = MyClass(a=0, b=4, params_dict={"p1": 1, "p2":-0.136, "p3":89})
myobject.set_param("p2", 20)
print(myobject.params_dict["p2"])
Update for the next comment: Well, you need to also make ParamDict
instance aware of MyClass
instance that contains it (or a method thereof it needs to call), e.g.:
class ParamsDict(dict):
def __init__(self, container, *args, **kwargs):
self._setparam_callback = container.setparam_callback
super().__init__(*args, **kwargs)
def __setitem__(self, key, value, /):
self._setparam_callback(key, value)
super().__setitem__(key, value)
class MyClass:
def __init__(self, a, b, params_dict=(("p1", 1), ("p2", -0.136))):
self.a = a
self.b = b
self.params_dict = ParamsDict(self, params_dict)
def setparam_callback(self, key, value):
print(f"Special action setting {key} to {value}")
myobject = MyClass(a=0, b=4, params_dict={"p1": 1, "p2":-0.136, "p3":89})
myobject.params_dict["p2"] = 20
If you want to hold set of key/value pairs (and you may not know what they are up front), dict
sounds like just a type for the job. I suspect you may be over-complicating things, simple:
myobject.set_param("p2", 20)
myobject.get_param("p2")
May look trivial(?), but gets the job done. It's also plenty obvious, which is generally a good thing.
Properties are not really going to help you all that much with what you've described. Somewhat similar appearance to expose data and "dynamically" handle access to attributes can be achieved by overloading getattr (and setattr for assignment), but again, simple is generally good.
How to get dictionary of sub-object using __dict__ on main object?
You can do something like this:
# Equivalent to your Engine class
class Foo:
def __init__(self, value):
self.v1 = value
# Equivalent to your Battery class
class Bar:
def __init__(self, value):
self.v2 = value
# Equivalent to your Wheels class
class Baz:
def __init__(self, value):
self.v3 = value
# Equivalent to your Car class
class FBB:
def __init__(self):
self.v = 0
self.foo = Foo(1)
self.bar = Bar(2)
self.baz = Baz(3)
fbb = FBB()
print({k: vars(v) if hasattr(v, '__dict__') else v for k, v in vars(fbb).items()})
# {'v': 0, 'foo': {'v1': 1}, 'bar': {'v2': 2}, 'baz': {'v3': 3}}
Note that this won't work if Bar
contains a reference to Baz
, for example. For that, you may consider creating a recursive function:
class Foo:
def __init__(self, value):
self.v1 = value
class Bar:
def __init__(self, value):
self.v2 = value
# Contains a reference to Baz!
self.baz = Baz(3)
class Baz:
def __init__(self, value):
self.v3 = value
class FBB:
def __init__(self):
self.v = 0
self.foo = Foo(1)
self.bar = Bar(2)
def full_vars(obj):
return {k: full_vars(v) if hasattr(v, '__dict__') else v for k, v in vars(obj).items()}
fbb = FBB()
print(full_vars(fbb))
# {'v': 0, 'foo': {'v1': 1}, 'bar': {'v2': 2, 'baz': {'v3': 3}}}
Related Topics
How to Group a List of Tuples/Objects by Similar Index/Attribute in Python
Python Postgres Psycopg2 Threadedconnectionpool Exhausted
Downloading with Chrome Headless and Selenium
What Is the Advantage of a List Comprehension Over a for Loop
How to Get JSON from Webpage into Python Script
Use .Corr to Get the Correlation Between Two Columns
How to Use Raw Socket in Python
It Is More Efficient to Use If-Return-Return or If-Else-Return
Valueerror: Numpy.Dtype Has the Wrong Size, Try Recompiling
Is There a "Not Equal" Operator in Python
What Is the Most Pythonic Way to Check If an Object Is a Number
Make a Post Request While Redirecting in Flask
How to Extract Parameters from a List and Pass Them to a Function Call
High-Precision Clock in Python