Merging Dictionary Value Lists in Python

Merging dictionary value lists in python

As a one-liner, with a dictionary comprehension:

new = {key: value + two[key] + [three[key]] for key, value in one.iteritems()}

This creates new lists, concatenating the list from one with the corresponding list from two, putting the single value in three into a temporary list to make concatenating easier.

Or with a for loop updating one in-place:

for key, value in one.iteritems():
    value.extend(two[key])
    value.append(three[key])

This uses list.extend() to update original list in-place with the list from two, and list.append() to add the single value from three.

Where you went wrong:

• your first attempt creates a new list with the values from one, two and three nested within rather than concatenating the existing lists. Your attempt to clean that up just copied those nested lists across.

• Your second attempt didn't work because the value in three is not a list so could not be concatenated. I created a new list just for that one value.

• Your last attempt should not have used list.append() in a generator expression, because you store the return value of that method, which is always None (its change is stored in v directly and the list doesn't need returning again).

Demo of the first approach:

>>> one={'a': [1, 2], 'c': [5, 6], 'b': [3, 4]}
>>> two={'a': [2.4, 3.4], 'c': [5.6, 7.6], 'b': [3.5, 4.5]}
>>> three={'a': 1.2, 'c': 3.4, 'b': 2.3}
>>> {key: value + two[key] + [three[key]] for key, value in one.iteritems()}
{'a': [1, 2, 2.4, 3.4, 1.2], 'c': [5, 6, 5.6, 7.6, 3.4], 'b': [3, 4, 3.5, 4.5, 2.3]}

Merge dictionaries in a list of dictionaries by combining the values

You can iterate over lst and create a dictionary out where the keys correspond to "id" values in the dicts in lst and the values are dicts. In each iteration, check if the value under "language" key is a list or not and append to the list if it's a list, create a list, if not. Finally, pass the values of out to a list constructor for the final outcome.

out = {}
for d in lst:
    if d['id'] in out:
        if isinstance(out[d['id']]['language'], list):
            out[d['id']]['language'].append(d['language'])
        else:
            out[d['id']]['language'] = [out[d['id']]['language'], d['language']]
    else:
        out[d['id']] = d
out = list(out.values())

Output:

[{'id': 1, 'name': 'Adel', 'language': ['С#', 'Python']},
 {'id': 5, 'name': 'Dora', 'language': ['С#', 'Java']},
 {'id': 6, 'name': 'Dars', 'language': 'Python'}]

How to merge dictionaries of dictionaries?

This is actually quite tricky - particularly if you want a useful error message when things are inconsistent, while correctly accepting duplicate but consistent entries (something no other answer here does..)

Assuming you don't have huge numbers of entries, a recursive function is easiest:

from functools import reduce

def merge(a, b, path=None):
    "merges b into a"
    if path is None: path = []
    for key in b:
        if key in a:
            if isinstance(a[key], dict) and isinstance(b[key], dict):
                merge(a[key], b[key], path + [str(key)])
            elif a[key] == b[key]:
                pass # same leaf value
            else:
                raise Exception('Conflict at %s' % '.'.join(path + [str(key)]))
        else:
            a[key] = b[key]
    return a

# works
print(merge({1:{"a":"A"},2:{"b":"B"}}, {2:{"c":"C"},3:{"d":"D"}}))
# has conflict
merge({1:{"a":"A"},2:{"b":"B"}}, {1:{"a":"A"},2:{"b":"C"}})

note that this mutates a - the contents of b are added to a (which is also returned). If you want to keep a you could call it like merge(dict(a), b).

agf pointed out (below) that you may have more than two dicts, in which case you can use:

reduce(merge, [dict1, dict2, dict3...])

where everything will be added to dict1.

Note: I edited my initial answer to mutate the first argument; that makes the "reduce" easier to explain

How to merge two list of dictionaries based on a value

You can keep track of the ids with another dict (or defaultdict to make things simpler). Then update the items in that dict as you iterate. In the end the dict's values will have your list.

from collections import defaultdict
d = defaultdict(dict)

a = [{'id': 1, 'name': 'a'}, {'id': 3, 'name': 'a'}]
b = [{'id': 1, 'city': 'b'}, {'id': 2, 'city': 'c'}, {'id': 3, 'city': 'd'}]

for item in a + b:
    d[item['id']].update(item)
list(d.values())

# [{'id': 1, 'name': 'a', 'city': 'b'},
#  {'id': 3, 'name': 'a', 'city': 'd'},
#  {'id': 2, 'city': 'c'}]

Note this will overwrite duplicate values other than id — so if you have two with id: 1 and two different cities, you will only get the last city.

Merge dictionaries with same key from two lists of dicts in python

Here is one of the approach:

a = {
   "name":"harry",
   "properties":[
      {
         "id":"N3",
         "status":"OPEN",
         "type":"energetic"
      },
      {
         "id":"N5",
         "status":"OPEN",
         "type":"hot"
      }
   ]
}
b = {
   "name":"harry",
   "properties":[
      {
         "id":"N3",
         "type":"energetic",
         "language": "english"
      },
      {
         "id":"N6",
         "status":"OPEN",
         "type":"cool"
      }
   ]
}

# Create dic maintaining the index of each id in resp dict
a_ids = {item['id']: index for index,item in enumerate(a['properties'])} #{'N3': 0, 'N5': 1}
b_ids = {item['id']: index for index,item in enumerate(b['properties'])} #{'N3': 0, 'N6': 1}

# Loop through one of the dict created
for id in a_ids.keys():
    # If same ID exists in another dict, update it with the key value
    if id in b_ids:
        b['properties'][b_ids[id]].update(a['properties'][a_ids[id]])
    # If it does not exist, then just append the new dict
    else:
        b['properties'].append(a['properties'][a_ids[id]])
        
        
print (b)

Output:

{'name': 'harry', 'properties': [{'id': 'N3', 'type': 'energetic', 'language': 'english', 'status': 'OPEN'}, {'id': 'N6', 'status': 'OPEN', 'type': 'cool'}, {'id': 'N5', 'status': 'OPEN', 'type': 'hot'}]}

Merge two lists into one dict that may contain several values for each key

Python does not have multidicts, so you have two options:

1. use multidicts from an existing library e.g. werkzeug's MultiDict, when initialised with lists of pairs, will associate multiple values to keys which are present multiple times (unlike dict which will only keep the last value)

   system_data_dict = werkzeug.datastructures.MultiDict(zip(system, instrument))
   system_data_dict.getlist('System A') # => ['Instrument 1', 'Instrument 2']
   

2. alternatively, do that by hand by using a regular loop, even if it's feasible using a comprehension (which I'm not sure about) it's going to look dreadful: use a defaultdict or the dict.setdefault method to define a dict mapping keys to lists and append values every time

   system_data_dict = {}
   for k, v in zip(system, instrument):
      system_data_dict.setdefault(k, []).append(v)
   system_data_dict['System A'] # => ['Instrument 1', 'Instrument 2']
   

---

Related Topics