Hitting Maximum Recursion Depth Using Pickle/Cpickle

Hitting Maximum Recursion Depth Using Pickle / cPickle

From the docs:

Trying to pickle a highly recursive data structure may exceed the maximum recursion depth, a RuntimeError will be raised in this case. You can carefully raise this limit with sys.setrecursionlimit().

Although your trie implementation may be simple, it uses recursion and can lead to issues when converting to a persistent data structure.

My recommendation would be continue raising the recursion limit to see if there is an upper bound for the data you are working with and the trie implementation you are using.

Other then that, you can try changing your tree implementation to be "less recursive", if possible, or write an additional implementation that has data persistence built-in (use pickles and shelves in your implementation). Hope that helps

Recursion Depth Exceeded, pickle and BeautifulSoup

try:

import sys
sys.setrecursionlimit(10000)

10 000 recursions should be enough
What happened is that somewhere someplace , something is calling itself over and over again. Each time, thats called one recursion. Python has a limit to prevent a program running infinitely. While this is usually a sign of an error, you can increase the limit to how you see fit, as your program might recurse for an unusually big amount.

RecursionError: maximum recursion depth exceeded while calling a Python object when using pickle.load()

Here's a fairly minimal class derived from collections.UserDict which performs the same trick that your problem object does. It's a dictionary which allows you to access its items either via normal dict syntax, or as attributes. I've thrown in a few print calls so we can see when the main methods get called.

import collections

class AttrDict(collections.UserDict):
    ''' A dictionary that can be accessed via attributes '''
    def __setattr__(self, key, value):
        print('SA', key, value)
        if key == 'data':
            super().__setattr__('data', value)
        else:
            self.data[key] = value

    def __getattr__(self, key):
        print('GA', key)
        if key in self.data:
            return self.data[key]
        else: 
            print('NOKEY')
            raise AttributeError

    def __delattr__(self, key):
        del self.data[key]

# test

keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])

output

SA data {}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}
GA zero
GA one
GA two
0 1 2 3

---

So far, so good. But if we try to pickle our d instance, we get RecursionError because of that __getattr__ which does the magic conversion of attribute access to key lookup. We can overcome that by providing the class with __getstate__ and __setstate__ methods.

import pickle
import collections

class AttrDict(collections.UserDict):
    ''' A dictionary that can be accessed via attributes '''
    def __setattr__(self, key, value):
        print('SA', key, value)
        if key == 'data':
            super().__setattr__('data', value)
        else:
            self.data[key] = value

    def __getattr__(self, key):
        print('GA', key)
        if key in self.data:
            return self.data[key]
        else: 
            print('NOKEY')
            raise AttributeError

    def __delattr__(self, key):
        del self.data[key]

    def __getstate__(self):
        print('GS')
        return self.data

    def __setstate__(self, state):
        print('SS')
        self.data = state

# tests

keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])

print('Pickling')
s = pickle.dumps(d, pickle.HIGHEST_PROTOCOL)
print(s)

print('Unpickling')
obj = pickle.loads(s)
print(obj)

output

SA data {}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}
GA zero
GA one
GA two
0 1 2 3
Pickling
GS
b'\x80\x04\x95D\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x08AttrDict\x94\x93\x94)\x81\x94}\x94(\x8c\x04zero\x94K\x00\x8c\x03one\x94K\x01\x8c\x03two\x94K\x02\x8c\x05three\x94K\x03ub.'
Unpickling
SS
SA data {'zero': 0, 'one': 1, 'two': 2, 'three': 3}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}

But what can we do to repair an existing class with this behaviour? Fortunately, Python allows us to easily add new methods to an existing class, even one that we obtain via importing.

import pickle
import collections

class AttrDict(collections.UserDict):
    ''' A dictionary that can be accessed via attributes '''
    def __setattr__(self, key, value):
        print('SA', key, value)
        if key == 'data':
            super().__setattr__('data', value)
        else:
            self.data[key] = value

    def __getattr__(self, key):
        print('GA', key)
        if key in self.data:
            return self.data[key]
        else: 
            print('NOKEY')
            raise AttributeError

    def __delattr__(self, key):
        del self.data[key]

# Patch the existing AttrDict class with __getstate__ & __setstate__ methods

def getstate(self):
    print('GS')
    return self.data

def setstate(self, state):
    print('SS')
    self.data = state

AttrDict.__getstate__ = getstate
AttrDict.__setstate__ = setstate

# tests

keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])

print('Pickling')
s = pickle.dumps(d, pickle.HIGHEST_PROTOCOL)
print(s)

print('Unpickling')
obj = pickle.loads(s)
print(obj)

This code produces the same output as the previous version, so I won't repeat it here.

Hopefully, this gives you enough info to repair your faulty object. My __getstate__ & __setstate__ methods only save and restore the stuff in the .data dictionary. To properly pickle your object, we may need to be a bit more drastic. For example, we may need to save and restore the instance's .__dict__ attribute, rather than just the .data attribute, which corresponds to the .meta attribute in your problem object.

python pickler - recursion depth exceeded

So, as @ExP said, pickler does depth-first pickling which causes recursion exceeded exception. Anyway, I found solution to this issue here bugs.python.org. That means for python 3.1 pickler works even on recursive data such as graphs for example.

There is also little less elegant solution which takes a lot more time to pickle some recursive data, but it's simple (just a matter of few lines of code). Link here.

As it seems, it's probably time to start slowly moving towards the python3. Hope that someone find this answer usefull.

Why does this pickle reach maximum recursion depth without recursion?

BeautifulSoup .string attributes aren't actually strings:

>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup('<div>Foo</div>')
>>> soup.find('div').string
    u'Foo'
>>> type(soup.find('div').string)
    bs4.element.NavigableString

Try using str(soup.find('div').string) instead and see if it helps. Also, I don't think Pickle is really the best format here. JSON is much easier in this case.

Python pickle error when dumping list of dictionaries

The error you are getting "maximum recursion depth exceeded" means that your the call stack is too deep. The depth of the call stack is basically how many functions were called from the initial call point(Something like your main function/loop) without returning.

So for example if from your initial call point you call func1(). In func1() the depth is currently 1. If func1(), before returning calls another function, the depth in that function will be 2.

To find out what the current limit is on the size of the call stack call sys.getrecursionlimit() and to change it, call sys.setrecursionlimit().

If the error is thrown when pickling the code, then chances are, the structure you are trying to pickle is has too many nested elements. This is because, the dump/dumps will recurse on containers in the structure being dumped. Meaning, if you're dumping a list, it will dump each element in the list which may themselves be a list or dict, so it will dump the elements those first and so on and so forth. If the elements are nested too deep(lists or dicts, which contain lists or dict, when contain lists or dicts ad infinitum) then at some point you will hit the maximum recursion depth.

The problem may be elsewhere, but without any further details that's the only thing that could be causing(if indeed that's where the error is thrown).

I have tried pickling a list of 1000 dicts(which are similar in structure to yours) and it worked just fine. Therefore either you're inadvertently building a structure that is too nested or that error is not getting thrown in the call to picke.dump()

Edit:

Try this function to see how nested the structure you're trying to dump is:

def nestedness(struct):
    if isinstance(struct, list):
        return max([0] + [nestedness(i) for i in struct]) + 1
    if isinstance(struct, dict):
        return max([0] + [nestedness(i) for i in struct.values()])+1
    return 1

If it fails with maximum recursion depth exceeded, keep raising it using sys.setrecursionlimit() until you get a proper answer. If your structure is just too nested see if you can find a way to make it less so by structuring it differently.

Edit: for posterity's sake, fixed nestedness so it handles empty dicts and sequences

---

Related Topics