How to "Perfectly" Override a Dict

How to perfectly override a dict?

You can write an object that behaves like a dict quite easily with ABCs (Abstract Base Classes) from the collections.abc module. It even tells you if you missed a method, so below is the minimal version that shuts the ABC up.

from collections.abc import MutableMapping

class TransformedDict(MutableMapping):
    """A dictionary that applies an arbitrary key-altering
       function before accessing the keys"""

    def __init__(self, *args, **kwargs):
        self.store = dict()
        self.update(dict(*args, **kwargs))  # use the free update to set keys

    def __getitem__(self, key):
        return self.store[self._keytransform(key)]

    def __setitem__(self, key, value):
        self.store[self._keytransform(key)] = value

    def __delitem__(self, key):
        del self.store[self._keytransform(key)]

    def __iter__(self):
        return iter(self.store)
    
    def __len__(self):
        return len(self.store)

    def _keytransform(self, key):
        return key

You get a few free methods from the ABC:

class MyTransformedDict(TransformedDict):

    def _keytransform(self, key):
        return key.lower()

s = MyTransformedDict([('Test', 'test')])

assert s.get('TEST') is s['test']   # free get
assert 'TeSt' in s                  # free __contains__
                                    # free setdefault, __eq__, and so on

import pickle
# works too since we just use a normal dict
assert pickle.loads(pickle.dumps(s)) == s

I wouldn't subclass dict (or other builtins) directly. It often makes no sense, because what you actually want to do is implement the interface of a dict. And that is exactly what ABCs are for.

How to properly subclass dict and override __getitem__ & __setitem__

What you're doing should absolutely work. I tested out your class, and aside from a missing opening parenthesis in your log statements, it works just fine. There are only two things I can think of. First, is the output of your log statement set correctly? You might need to put a logging.basicConfig(level=logging.DEBUG) at the top of your script.

Second, __getitem__ and __setitem__ are only called during [] accesses. So make sure you only access DictWatch via d[key], rather than d.get() and d.set()

Override dict() on class

dict can be called with an iterable of pairs, so if you design your __iter__ to return an iterable of tuples, your example works as you'd like:

class Foo:
    def __iter__(self):
        yield from {
            'this': 'is',
            'a': 'dict'
        }.items()

dict(Foo())
{'a': 'dict', 'this': 'is'}

---

If you want your class to behave like a python dictionary, in that iterating over an instance iterates over its keys, you can implement the interface defined by abc.Mapping.

You can do this either by implementing __getitem__, __iter__, and __len__, and inheriting from abc.Mapping, or by implementing all of __getitem__, __iter__, __len__ __contains__, keys, items, values, get, __eq__, and __ne__.

python: override a single dict key based on a separate dict

Just try to replace it. This avoids one unnecessary hashing (compared to using if 'day' in query:) or looping over the dictionary and follows Pythons EAFP principle:

try:
    query['day'] = day_mapping[query['day']]
except KeyError:
    pass

Override a dict with numpy support

The problem is in the np.array constructor step. It digs into its inputs trying to create a higher dimensional array.

In [99]: basic={'test.field':'test'}

In [100]: eb=Extendeddict(basic)

In [104]: eba=np.array([eb],object)
<keys: 0,[0]>
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-104-5591a58c168a> in <module>()
----> 1 eba=np.array([eb],object)

<ipython-input-88-a7d937b1c8fd> in __getitem__(self, key)
     11         keys = self._keytransform(key);print key;print keys
     12         if len(keys) == 1:
---> 13             return self._store[key]
     14         else:
     15             key1 = '.'.join(keys[1:])

KeyError: 0 

But if I make an array, and assign the object it works fine

In [105]: eba=np.zeros((1,),object)

In [106]: eba[0]=eb

In [107]: eba
Out[107]: array([{'test': {'field': 'test'}}], dtype=object)

np.array is a tricky function to use with dtype=object. Compare np.array([[1,2],[2,3]],dtype=object) and np.array([[1,2],[2]],dtype=object). One is (2,2) the other (2,). It tries to make a 2d array, and resorts to 1d with list elements only if that fails. Something along that line is happening here.

I see 2 solutions - one is this round about way of constructing the array, which I've used in other occasions. The other is to figure out why np.array doesn't dig into dict but does with yours. np.array is compiled, so that may require reading tough GITHUB code.

---

I tried a solution with f=np.frompyfunc(lambda x:x,1,1), but that doesn't work (see my edit history for details). But I found that mixing an Extendeddict with a dict does work:

In [139]: np.array([eb,basic])
Out[139]: array([{'test': {'field': 'test'}}, {'test.field': 'test'}], dtype=object)

So does mixing it with something else like None or an empty list

In [140]: np.array([eb,[]])
Out[140]: array([{'test': {'field': 'test'}}, []], dtype=object)

In [142]: np.array([eb,None])[:-1]
Out[142]: array([{'test': {'field': 'test'}}], dtype=object)

This is another common trick for constructing an object array of lists.

It also works if you give it two or more Extendeddict with different lengths

np.array([eb, Extendeddict({})]). In other words if len(...) differ (just as with mixed lists).

Subclass dict __getitem__ while maintaining original class type

How about your MyDict just be a proxy for the dict:

class MyDict(object):
    def __init__(self, data={}):
        self.data = data

    def __getitem__(self, key):
        if key == 'b':
            print('Found "b"')
        return MyDict(self.data.__getitem__(key))

    def __setitem__(self, key, value):
        return self.data.__setitem__(key, value)

    def __repr__(self):
        return self.data.__repr__()

    # add more __magic__ methods as you wish

shallow_dict = MyDict({
    'b': 'value'
})

x = shallow_dict['b']

deep_dict = MyDict({
    'a': {
        'b': 'value'
    }
})

x = deep_dict['a']['b']

# assignment
deep_dict['a']['a'] = {'b': 'here'}
deep_dict['a']['a']['b']
print(deep_dict)

OUTPUT:

Found "b"
Found "b"
Found "b"
{'a': {'b': 'value', 'a': {'b': 'here'}}}

As you can see when you get the self.data inside __getitem__ it just pass the result of self.data.__getitem__ by reference to a new MyDict object.

In method call args, how to override keyword argument of unpacked dict?

You can use the built-in dict type for that purpose. It accepts another dict as argument and additional key-value pairs as keyword arguments (which have precedence over the values in the other dict).

Thus you can create an updated dictionary via dict(template_vars, a=1).

You can unfold this dict as keyword arguments: func(**dict(...)).

Like that there is no need to change the signature of your function and you can update/add as many key-value pairs as you want.

---

Related Topics