Python 3.6 pickling custom procedure

I have some objects of class A which has its own method to be pickled, call it custom_module.customPickle(A) which takes an instance of A and returns a serialization string. I also have list of objects each of class B that contains A .

I need to pickle the list, but pickling A gives some error difficult to solve. However, A has its own method to be pickled.

I can implement the __reduce__() method in class B so that it calls custom_module.customPickle(A) . But how can I do this so that pickle is able to serialize B efficiently?

---

Object A is a music21.stream and object B is a custom object. The custom serialization function is music21.converter.freezeStr(streamObj, fmt=None) and the unpickle function should be music21.converter.thawStr(strData)

---

You can use the copyreg module to register custom functions for pickling and unpickling; the function you register acts like a __reduce__ method on the class.

If you return a tuple of (unpickle_function, state) , then the registered unpickle_function callable will be called to unpickle it again, with state as the argument, so you can use your music21.converter.thawStr() function there:

import copyreg
import music21.converter
import music21.stream


def pickle_music21_stream(stream_obj):
    return music21.converter.thawStr, (music21.converter.freezeStr(stream_obj),)

copyreg.pickle(music21.stream.Stream, pickle_music21_stream)

(the constructor argument to copyreg is ignored in recent Python versions)

This registers a global handler for those objects. You can also use a dispatch table per pickler, see [*Dispatch Tables on how you'd register one.

Now, when pickling, when encountering any instances of Stream the handle_stream() function is used to produce a serialisation, and the thawStr() function will be used to unpickle that data again.

However, the music21.converter functions use pickle themselves. They effectively pack and clean up the stream, and then pickle the resulting Stream instance. This will then call the custom handler, and you have an infinite loop.

The work-around is to use a custom dispatch table to handle pickling and unpickling. Avoid using copyreg in this case, as it sets a global hook that'll be called recursively each time a Stream object is being pickled.

Your own pickle infrastructure needs to use a custom pickler:

import copyreg
import io
import pickle
import music21.converter
import music21.stream


def pickle_music21_stream(stream_obj):
    return music21.converter.thawStr, (music21.converter.freezeStr(stream_obj),)


def dumps(obj):
    f = io.BytesIO()
    p = pickle.Pickler(f)
    p.dispatch_table = copyreg.dispatch_table.copy()
    p.dispatch_table[music21.stream.Stream] = pickle_music21_stream
    p.dump(obj)
    return f.getvalue()


def loads(data):
    return pickle.loads(data)  # hook is registered in the pickle data

Here the custom function is only called when a Stream instance is found in your own data structure. The music21 routines use the global pickle.dumps() and pickle.loads() functions and won't use the same hook.

链接地址: http://www.djcxy.com/p/64832.html

上一篇: Joblib无法正确拔除类

下一篇: Python 3.6酸洗自定义程序