3 ms·
Why this over serialization free formats like CapnProto and Flatbuffers? If you want it to be compact, send it through zstd (with a custom dictionary). I do re
by seg_lol 1y ago
Why this over serialization free formats like CapnProto and Flatbuffers? If you want it to be compact, send it through zstd (with a custom dictionary).
I do really like that is broad support out of the box and looks easy to use.
For Python I still prefer using dill since it handles code objects.
https://github.com/uqfoundation/dill https://github.com/uqfoundation/dill
- chaokunyang 1y agoApache Fory is also a drop-in replacement for pickle/cloudpickle, you can use it to serialize code object such as local function/Classes too. https://github.com/apache/fory/tree/main/python#serialize-local-functionslambdas https://github.com/apache/fory/tree/main/python#serialize-lo... When serializing code objects, pyfory is 3× higher compression ratio compared cloudpickle And pyfory also provide extra security audit capability to avoid maliciously deserialization data attack.
- chaokunyang 1y agoI also did a benchmark with dill, it shows that: fory is 20~40X faster and up to 7x higher compression ratio compared to dill. I don't dive into dill to see how it works. Here is my benchmark code: https://github.com/chaokunyang/python_benchmarks https://github.com/chaokunyang/python_benchmarks