4 ms·

Have you installed both protocol buffers and the fast-python-pb module? Feel free to email me: robbyw@(the-company-mentioned-in-the-title).com

• sigil 16y ago
  Thanks Robby, got the benchmark working, was trying to do a homedir install of fast-python-pb earlier. I added a couple more tests to the benchmark, here are the results: JSON 3.57209396362 Protocol Buffer (fast-python-pb) 0.325706005096 Protocol Buffer (native) 4.83730196953 Protocol Buffer (lwpb) 0.32919216156 cPickle 0.837985038757 As you can see, lwpb and fast-python-pb are neck and neck. And I should point out that lwpb isn't using C++ codegen at all, just the compiled schema in the .proto file. Of course, if completeness of implementation was the critical thing, you'd probably want to stay closer to google's official implementation. There's a lot of the google implementation that I never use though, like the RPC stuff. Also notable that both lwpb and fast-python-pb outperform cPickle by almost 3x. It would be interesting to know why a portable, cross-language serialization format beats out the language-specific one. Here's a fork with the patched benchmark code: https://github.com/acg/fast-python-pb https://github.com/acg/fast-python-pb
  • btilly 16y ago
    It would be interesting to know why a portable, cross-language serialization format beats out the language-specific one. Odds are that it is because the language-specific one supports features that the portable one does not. A feature I'd be particularly suspicious of is, "Did we already encounter this data structure and serialize it?" Supporting that feature means tracking a LOT of information during the serialization process, whether or not you get to use it.
  • deleted 16y ago
    [deleted]
  • ot 16y ago
    > It would be interesting to know why a portable, cross-language serialization format beats out the language-specific one. Because protobuf parses messages with a fixed schema in a very structured format. Pickle, OTOH, is an interpreted bytecode microlanguage used to describe arbitrary python objects (for instance, pickle can call python functions: http://nadiana.com/python-pickle-insecure http://nadiana.com/python-pickle-insecure) Also, pickle supports references (so if an object is referenced two times in the same serialization stream, it is serialized only once) and this has a cost at serialization time (need to keep the set of seen references)
    • sigil 16y ago
      Perl's Storable can also serialize code references which get eval'ed during deserialization, and can also serialize multiple references once, though you have to be more explicit about that. And Storable is still 2x-3x faster than the already quite fast JSON::XS. It would seem the bytecode interpreter architecture in Pickle is the limiting factor. If anybody has some good profiling data on Pickle though, I'd love to see it.
  • olt 16y ago
    (c)pickle uses a string protocol by default which is the most portable (even across Python versions and platforms). You need to specify another protocol version for best performance. See http://docs.python.org/library/pickle.html#data-stream-format http://docs.python.org/library/pickle.html#data-stream-forma... The times should be similar to lwpb then. Edit2: Oh, and for JSON you should use http://pypi.python.org/pypi/simplejson http://pypi.python.org/pypi/simplejson
    • sigil 16y ago
      Yup, simplejson is much faster than the standard json. JSON 3.56521892548 SimpleJSON 0.727998971939 Protocol Buffer (fast) 0.38397192955 Protocol Buffer (standard) 4.86640501022 Protocol Buffer (lwpb) 0.323328971863 cPickle 0.811990976334
      • indigoviolet 16y ago
        I think you'll find py-yajl to be faster than any of the other Python JSON modules: https://github.com/rtyler/py-yajl https://github.com/rtyler/py-yajl