Python Multiprocessing Picklingerror: Can't Pickle ≪Type 'Function'≫

Python multiprocessing PicklingError: Can't pickle <type 'function'>

Here is a list of what can be pickled. In particular, functions are only picklable if they are defined at the top-level of a module.

This piece of code:

import multiprocessing as mp

class Foo():
    @staticmethod
    def work(self):
        pass

if __name__ == '__main__':   
    pool = mp.Pool()
    foo = Foo()
    pool.apply_async(foo.work)
    pool.close()
    pool.join()

yields an error almost identical to the one you posted:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 315, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

The problem is that the pool methods all use a mp.SimpleQueue to pass tasks to the worker processes. Everything that goes through the mp.SimpleQueue must be pickable, and foo.work is not picklable since it is not defined at the top level of the module.

It can be fixed by defining a function at the top level, which calls foo.work():

def work(foo):
    foo.work()

pool.apply_async(work,args=(foo,))

Notice that foo is pickable, since Foo is defined at the top level and foo.__dict__ is picklable.

python multiprocessing Can't pickle <type 'function'>

Python's multiprocessing module can not deal with functions/methods which cannot be pickled, which means you cannot use class or instance methods without a lot of hassle. I would recommend to use multiprocess, which uses dill for serialization instead of pickle, and can deal with class or instance methods.

As far as I know, the interface is exactly the same as the one used in multiprocessing, so you can use it as a drop-in replacement.

See also https://stackoverflow.com/a/21345423/1170207

Can't pickle <type 'instancemethod'> when using multiprocessing Pool.map()

The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it "easy" or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_reg standard library method.

For example, Steven Bethard's contribution to this thread (towards the end of the thread) shows one perfectly workable approach to allow method pickling/unpickling via copy_reg.

Python multiprocessing basic - Can't pickle local object and Ran out of input

Multiprocessing uses pickle to serialize and transfer data between sub-processes.

Pickle cannot serialize local (inner) functions. Try setting your target to a function visible from the file's namespace.

For example:

import multiprocessing


def abc(self):

    try:
        """Some
            logic
            here"""


    except Exception as E:
        print('Error : ', E)

def func1(sfunc1):

    """some
    logic
    here"""

def func2(sfunc2):

    """some
    logic
    here"""

def check(stk):

    p1 = multiprocessing.Process(target=func1, args=s1)  # s1 and s2 is a List
    p2 = multiprocessing.Process(target=func2, args=s2)
    p1.start()
    p2.start()
    p1.join()
    p2.join()

if __name__ == "__main__":  # Make sure only 1 process runs this.
    check(Symbols)

Make sure s1, s2, and Symbols are defined.

---

Related Topics