Unpickling a Python 2 Object with Python 3

Unpickling a python 2 object with python 3

You'll have to tell pickle.load() how to convert Python bytestring data to Python 3 strings, or you can tell pickle to leave them as bytes.

The default is to try and decode all string data as ASCII, and that decoding fails. See the pickle.load() documentation:

Optional keyword arguments are fix_imports, encoding and errors, which are used to control compatibility support for pickle stream generated by Python 2. If fix_imports is true, pickle will try to map the old Python 2 names to the new names used in Python 3. The encoding and errors tell pickle how to decode 8-bit string instances pickled by Python 2; these default to ‘ASCII’ and ‘strict’, respectively. The encoding can be ‘bytes’ to read these 8-bit string instances as bytes objects.

Setting the encoding to latin1 allows you to import the data directly:

with open(mshelffile, 'rb') as f:
    d = pickle.load(f, encoding='latin1') 

but you'll need to verify that none of your strings are decoded using the wrong codec; Latin-1 works for any input as it maps the byte values 0-255 to the first 256 Unicode codepoints directly.

The alternative would be to load the data with encoding='bytes', and decode all bytes keys and values afterwards.

Note that up to Python versions before 3.6.8, 3.7.2 and 3.8.0, unpickling of Python 2 datetime object data is broken unless you use encoding='bytes'.

Passing a python object from a python2 script to python3 script

The binary representation of dictionaries, and in fact objects in general, has changed quite a bit between Python 2 and Python 3. That means that you will not be able to use the objects directly between versions of Python. Some form of serialization to a mutually acceptable format will be necessary.

The JSON format you are using fits this criterion. I would recommend sticking with it as it is flexible, human readable and quite general.

If space becomes an issue, you can look into pickling as an alternative, but I would not recommend it as a first choice. While pickle.load can understand Python 2 data just fine, there are many caveats to keep in mind with different interpretations of datatypes between Python versions, especially bytes: Unpickling a python 2 object with python 3

Here is a resource that may help you with some common errors from unpicking on the Python 3 side: TypeError: a bytes-like object is required, not 'str' when opening Python 2 Pickle file in Python 3

Unpickling classes from Python 3 in Python 2

This problem is Python issue 3675. This bug is actually fixed in Python 3.11.

If we import:

from lib2to3.fixes.fix_imports import MAPPING

MAPPING maps Python 2 names to Python 3 names. We want this in reverse.

REVERSE_MAPPING={}
for key,val in MAPPING.items():
    REVERSE_MAPPING[val]=key

We can override the Unpickler and loads

class Python_3_Unpickler(pickle.Unpickler):
    """Class for pickling objects from Python 3"""
    def find_class(self,module,name):
        if module in REVERSE_MAPPING:
            module=REVERSE_MAPPING[module]
        __import__(module)
        mod = sys.modules[module]
        klass = getattr(mod, name)
        return klass

def loads(str):
    file = pickle.StringIO(str)
    return Python_3_Unpickler(file).load()  

We then call this loads instead of pickle.loads.

This should solve the problem.

---

Related Topics