Python: Can't Pickle Type X, Attribute Lookup Failed

Python: Can't pickle type X, attribute lookup failed

Yes, the fact that it's a class member is a problem:

>>> class Foo():
...     Bar = namedtuple('Bar', ['x','y'])
...     def baz(self):
...         b = Foo.Bar(x=2, y=3)
...         print(type(b))
...
>>> a = Foo()
>>> a.baz()
<class '__main__.Bar'>

The problem is that when namedtuple() returns a type object, it isn't aware of the fact that it's being assigned to a class member - and thus, it tells the type object that its type name should be __main__.Bar, even though it should really be __main__.Foo.Bar.

Attribute lookup Code on builtins failed error happens when a user defined object returned from map operation in Pyspark

Based on the answers and comments here : Python: Can't pickle type X, attribute lookup failed

It seems you need to make Base class its own module and then it will work.
So here is Base.py

class Base:
    def __init__(self,line):
        self.line = line

And mainscript.py

from pyspark import SparkContext, SQLContext
from Base import Base

sc = SparkContext('local')
sqlContext = SQLContext(sc)

def process_line_baseclass(line):
    return Base(line)

input_path = 'path/to/inputfile'
samples = sc.textFile(input_path).map(process_line_baseclass)

print(samples.take(1))

Output :

[<Base.Base object at 0x7fc2188e64a8>]

Python multiprocessing PicklingError: Can't pickle type 'function'

Here is a list of what can be pickled. In particular, functions are only picklable if they are defined at the top-level of a module.

This piece of code:

import multiprocessing as mp

class Foo():
    @staticmethod
    def work(self):
        pass

if __name__ == '__main__':   
    pool = mp.Pool()
    foo = Foo()
    pool.apply_async(foo.work)
    pool.close()
    pool.join()

yields an error almost identical to the one you posted:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 315, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

The problem is that the pool methods all use a mp.SimpleQueue to pass tasks to the worker processes. Everything that goes through the mp.SimpleQueue must be pickable, and foo.work is not picklable since it is not defined at the top level of the module.

It can be fixed by defining a function at the top level, which calls foo.work():

def work(foo):
    foo.work()

pool.apply_async(work,args=(foo,))

Notice that foo is pickable, since Foo is defined at the top level and foo.__dict__ is picklable.

pickle.PicklingError: Can't pickle class 'module': attribute lookup module on builtins failed

multiprocessing is not a substitute for threads.

Code running in processes created with multiprocessing runs in a separate process from the process which created it. As such, it cannot access objects related to the Tk GUI; those objects are only valid in the parent process.

If you need to terminate a thread, use a condition variable to notify it that it's time to stop.

Shelve: Can't pickle class 'method': attribute lookup builtins.method failed

The first problem is caused by this line in your entity class:

self.current_action = {'action': self.wait, 'cost': self.speed}

current_action contains references to bound methods, which can't be pickled. You could use __getstate__ and __setstate__ do change your pickling behaviour, so you don't pickle the method but it's name (or name and arguments in the case where have assigned a partial object) when pickling, and restore that values at unpickling time.

I'm not really sure about your second problem, but if it only happens when you run it inside a debugger, it's probably an issue with how the debuger loads the __main__ module. You could try to move your __main__ block into another module and import and lounch your code from there. Sometimes that can fix problems like these.

Can't pickle type 'function': attribute lookup __builtin__.function failed

Where can I print out the object that is giving issues from this error so that it will appear in the Django error page?

The short answer - without recompiling cPickle, you can't.

Longer answer: This is the piece of code that raises the exception:

root $ grep -Hra "attribute lookup" /usr/lib64/ 2>/dev/null | grep -a failed
/usr/lib64/python2.7/lib-dynload/cPickle.so:H�H���P0H�5zM H�=1��M��H��H�ZM �����H�=�1�H���M��H��H��M �����H�4M H�5~H���H���������H�M H�5tH���H���������H��L H�5�H���qH�����d���H�M H�5NH���SH�����F���H�dM H�5bH���5H�����(���H�=X1��O��H�HC H�5I H�=U1�A��H��L �vH��H��I�������H���RK��H�=.H����J��H�5$H��H��H�D$�G��H�D$H��tH�H��H��H��gH�DL�ttH�qH�5nH�=kH��1��XK��H�5bH��H��I����F��H�5\L��H���F��H��tH�EH��H��H�E��M�������I�$H��H��I�$�t���I�DL���P0�d���f�H�EH��H��H�E�F���H�H���P0�7���@H�|$H��P0�����H�H���P0�����H�H���P0�k�����UH��SH�H�H: H���tH�;: H���H�H���u�H�[��H��M��H��attribute deletion is not supportedunsupported pickle protocol: %dargument must have 'read' and 'readline' attributespickle protocol %d asked for; the highest available protocol is %dargument must have 'write' attributeGlobal and instance pickles are not supported.Attempt to getvalue() a non-list-based picklerUnexpected data in internal listBINSTRING pickle has negative byte countno int where int expected in memoCan't pickle %s: import of module %s failedCan't pickle %s: attribute lookup %s.%s failedCan't pickle %s: it's not the same object as %s.%sCan't pickle %s: extension code %s isn't an integerCan't pickle %s: extension code %ld is out of rangecould not convert string to intLONG pickle has negative byte countcould not convert string to floatBINUNICODE pickle has negative byte countunregistered extension code %ld_inverted_registry[%ld] isn't a 2-tuple of stringsA load persistent id instruction was encountered,

If you look close enough, there's a piece that says

Can't pickle %s: attribute lookup %s.%s failed

Now, if you download Python sources, you can easily find the piece of code responsible for raising the exception in the function static int save_global(Picklerobject *self, PyObject *args, PyObject *name) of ./Modules/cPickle.c:

klass = PyObject_GetAttrString(mod, name_str);
if (klass == NULL) {
    cPickle_ErrFormat(PicklingError,
                      "Can't pickle %s: attribute lookup %s.%s "
                      "failed",
                      "OSS", args, module, global_name);
    goto finally;
}

So, the best you can do to debug this error is to format the string differently (probably providing PyString_AS_STRING((PyStringObject *)name), recompile and install the modified version of Python.

Yeah, I know that's too bad. I just had the same problem myself.

PicklingError: Can't pickle class 'module': attribute lookup module on builtins failed

From the documentation:

It is not recommended to use pickle or cPickle to save a Keras model.

You can use model.save(filepath) to save a Keras model into a single
HDF5 file which will contain:

• the architecture of the model, allowing to re-create the model
• the weights of the model
• the training configuration (loss, optimizer)
• the state of the optimizer, allowing to resume training exactly where you left off. You can then use keras.models.load_model(filepath)
  to reinstantiate your model.

To save your model, you'd need to call model.save:

model.save('model.h5')  # creates a HDF5 file 'model.h5'

Similarly, loading the model is done like this:

from keras.models import load_model
model = load_model('model.h5')

---

Related Topics