What Do I Do When I Need a Self Referential Dictionary

What do I do when I need a self referential dictionary?

No fear of creating new classes -
You can take advantage of Python's string formating capabilities
and simply do:

class MyDict(dict):
   def __getitem__(self, item):
       return dict.__getitem__(self, item) % self

dictionary = MyDict({

    'user' : 'gnucom',
    'home' : '/home/%(user)s',
    'bin' : '%(home)s/bin' 
})

print dictionary["home"]
print dictionary["bin"]

Python self referential dictionary with class

self doesn't just magically appear; Python functions are wrapped in methods when accessed via an instance (through invoking the function as a descriptor), and self is then passed in to the function as a first argument.

For your lambdas, you'll need to pass self in explicitly:

class ParamDict(dict):
    def __getitem__(self, key):
        val = dict.__getitem__(self, key)
        return callable(val) and val(self) or val

params = ParamDict({
    'a': 5,
    'b': lambda self: 2**self['a']
    })

Self-referential dictionary comprehension

In general you won't be able to reference the dictionary itself in the comprehension, because the name won't get assigned to the resulting dictionary until the comprehension is completed, so you'll have to settle for predefining the dictionary* and utilize mutating methods of the existing dictionary.

Since you're iterating over the input list, you'll need to update the existing dictionary with the new values whenever you come across the key. And as you can't use an assignment in the dictionary comprehension, you'll want to use the dict.update() method (or __setitem__ or setdefault). That method always returns None, so you can utilize that to achieve the desired side effect in a number of different places inside the dictionary comprehension.

In particular, any filtering condition clause will be executed, so you can use that. Alternatively, expr or value will evaluate the expression, which will always return None, and since that's falsey the whole expression evaluates to value, so you can place that expression in the key or value. That gives us the following possibilities:

With the side effect in the filter clause:

d = {}
d = {k: d[k] for k, *vals in x if d.update({k: d.get(k, []) + vals}) is None}

With the side effect in a expr or key expression:

d = {}
d = {d.update({k: d.get(k, []) + vals}) or k: d[k] for k, *vals in x}

With the side effect in a expr or value expression:

d = {}
d = {k: d.update({k: d.get(k, []) + vals}) or d[k] for k, *vals in x}

* Using assignment expressions (Python 3.8+), you can predefine the dictionary inside the comprehension itself with this abomination:

d = {k: d.update({k: d.get(k, []) + vals}) or d[k] for i, (k, *vals) in enumerate(x) if i or not (d := {})}

This uses enumerate() to detect when you're on the first iteration, in which case an assignment expression can construct the dictionary that is used in the rest of the comprehension. After the first iteration, the assignment expression is not evaluated again so d doesn't get reassigned in the course of the evaluation.

Note: Obviously, all of the methods shown in this answer are terrible. Side-effects inside comprehensions are unnecessary, unexpected, confusing, and in one word, silly. Do not use this code. But it is fun to see what's possible!

Dynamic Self Referencing Dictionary in Python

Something like this perhaps? Tested with Python 2.7.

from UserDict import UserDict

class Barn(UserDict):
    '''Barn automatically sets correct values for chickens.
    TBD: What to do if chickens get changed?'''
    def __init__(self, *args, **kw):
        UserDict.__init__(self, *args, **kw)
        pigs = self.data.get('pigs', None)
        if pigs is not None:
            self.data['chicken'] = pigs * 2

    def __setitem__(self, name, value):
        if name == 'pigs':
            self.data['chicken'] = value * 2
        self.data[name] = value

if __name__ == '__main__':
    b = Barn(cows=1, pigs=2)
    print b
    b['pigs'] = 3
    print b

Running that should produce something like this:

{'cows': 1, 'chicken': 4, 'pigs': 2}
{'cows': 1, 'chicken': 6, 'pigs': 3}

Is there any usage of self-referential lists or circular reference in list, eg. appending a list to itself

The use case is that Python is a dynamically typed language, where anything can reference anything, including itself.

List elements are references to other objects, just like variable names and attributes and the keys and values in dictionaries. The references are not typed, variables or lists are not restricted to only referencing, say, integers or floating point values. Every reference can reference any valid Python object. (Python is also strongly typed, in that the objects have a specific type that won't just change; strings remain strings, lists stay lists).

So, because Python is dynamically typed, the following:

foo = []
# ...
foo = False

is valid, because foo isn't restricted to a specific type of object, and the same goes for Python list objects.

The moment your language allows this, you have to account for recursive structures, because containers are allowed to reference themselves, directly or indirectly. The list representation takes this into account by not blowing up when you do this and ask for a string representation. It is instead showing you a [...] entry when there is a circular reference. This happens not just for direct references either, you can create an indirect reference too:

>>> foo = []
>>> bar = []
>>> foo.append(bar)
>>> bar.append(foo)
>>> foo
[[[...]]]

foo is the outermost [/]] pair and the [...] entry. bar is the [/] pair in the middle.

There are plenty of practical situations where you'd want a self-referencing (circular) structure. The built-in OrderedDict object uses a circular linked list to track item order, for example. This is not normally easily visible as there is a C-optimised version of the type, but we can force the Python interpreter to use the pure-Python version (you want to use a fresh interpreter, this is kind-of hackish):

>>> import sys
>>> class ImportFailedModule:
...     def __getattr__(self, name):
...         raise ImportError
...
>>> sys.modules["_collections"] = ImportFailedModule()  # block the extension module from being loaded
>>> del sys.modules["collections"]  # force a re-import
>>> from collections import OrderedDict

now we have a pure-python version we can introspect:

>>> od = OrderedDict()
>>> vars(od)
{'_OrderedDict__hardroot': <collections._Link object at 0x10a854e00>, '_OrderedDict__root': <weakproxy at 0x10a861130 to _Link at 0x10a854e00>, '_OrderedDict__map': {}}

Because this ordered dict is empty, the root references itself:

>>> od._OrderedDict__root.next is od._OrderedDict__root
True

just like a list can reference itself. Add a key or two and the linked list grows, but remains linked to itself, eventually:

>>> od["foo"] = "bar"
>>> od._OrderedDict__root.next is od._OrderedDict__root
False
>>> od._OrderedDict__root.next.next is od._OrderedDict__root
True
>>> od["spam"] = 42
>>> od._OrderedDict__root.next.next is od._OrderedDict__root
False
>>> od._OrderedDict__root.next.next.next is od._OrderedDict__root
True

The circular linked list makes it easy to alter the key ordering without having to rebuild the whole underlying hash table.

Reference a dictionary within itself

Why not update the dictionary:

my_dict = {'root': '/var/tmp'}
my_dict.update({'file': os.path.join(my_dict.get('root'), 'file')})

Don't use dict as a name. You may need the real dict builtin later on.

Is there a pythonic way of referring to the current object (self-reference) we are declaring with (some) Pythons built-in types?

Python provides no way to refer to an object under construction by a literal or a display*. You can (ab)use the assignment expression in Python 3.8 or later to simulate this:

a_dictionary = {
    "key_1": (x := "value_1"),
    "key_2": (y := "value_2"),
    "key_3": x + "_" + y 
}

It requires some planning ahead, as you are not referring to a key value directly, rather a pre-defined variable. Notice that x and y remain in scope after the assignment to a_dictionary, so this is just a questionable equivalent of

x = "value_1"
y = "value_2"
a_dictionary = {
    "key_1": x,
    "key_2": y,
    "key_3": x + "_" + y
}

A custom class would really be more appropriate:

class Thing:
    def __init__(self, v1, v2):
        self.key_1 = v1
        self.key_2 = v2
        self.key_3 = v1 + "_" + v2

a_thing = Thing("value_1", "value_2")

A display is a construct like a literal, but could contain non-literal references. For example, list displays include [1, x, y] and [int(x) for x in foo].

Pythonic way to walk a self-referential dictionary

To "walk" the dictionary, just do the lookups in a loop until there are no more:

>>> def walk(d, val):
        while val in d:
            val = d[val]
        return None if val == '-' else val

>>> d = {'1': '-', '0': '6', '3': '1', '2': '3', '4': '5', '6': '9'}
>>> print {k: walk(d, k) for k in d}
{'1': None, '0': '9', '3': None, '2': None, '4': '5', '6': '9'}