4 ms·

Google's documentation claims that protocol buffers are designed to be fast so I tested them out and I also found that their Python implementation was too slow

by andres 17y ago

Google's documentation claims that protocol buffers are designed to be fast so I tested them out and I also found that their Python implementation was too slow to use. I posted a question to the Discussion Forum and found out that it's a known (but undocumented!!!) problem: http://groups.google.com/group/protobuf/browse_thread/thread/2f57569563a6a476/53ed0cf5cf5ee1ff?lnk=gst&q=andres#53ed0cf5cf5ee1ff http://groups.google.com/group/protobuf/browse_thread/thread... Recently I tried out keyczar, Google's crypto toolkit and found that their Python implementation was too slow by a factor of 100X because they are using a slow random string generator: http://groups.google.com/group/keyczar-discuss/browse_thread/thread/8ed3a503dad4b6b7 http://groups.google.com/group/keyczar-discuss/browse_thread... It's hard to see how both of these problems could have passed internal benchmarking and made it into live code. Maybe Google isn't using its own Python implementations internally.

lacker 17y ago
The internal standard for when speed of protocol buffer operations matters is to use the C++ protocol buffer code but use swig to wrap it for python. Unfortunately, this is really easy internally due to protocol buffer integration with our build system, but not easy externally since most people don't have a build phase for their python projects.
jasonwatkinspdx 17y ago
Large systems tend to push you towards tradeoffs other people wouldn't make. Scalability over efficiency. Bisection bandwidth is the most scarce resource in a large cluster so I imagine protocol buffers have had much more attention paid to saving bytes than to encoding/decoding speed. Latency critical stuff probably doesn't use python anyhow.
- haberman 17y ago
  Protobufs are actually pretty fast to encode and decode (in the neighborhood of 200-300MB/s on my core2 desktop, when using the C++ bindings). It's just the Python implementation that is slow. I'm working on a Python implementation that will be much faster. It's really unfortunately that Protocol Buffers are getting a bad rap due to the current Python implementation.

by andres 17y ago

lacker 17y ago

The internal standard for when speed of protocol buffer operations matters is to use the C++ protocol buffer code but use swig to wrap it for python. Unfortunately, this is really easy internally due to protocol buffer integration with our build system, but not easy externally since most people don't have a build phase for their python projects.

jasonwatkinspdx 17y ago

Large systems tend to push you towards tradeoffs other people wouldn't make. Scalability over efficiency. Bisection bandwidth is the most scarce resource in a large cluster so I imagine protocol buffers have had much more attention paid to saving bytes than to encoding/decoding speed. Latency critical stuff probably doesn't use python anyhow.

haberman 17y ago
Protobufs are actually pretty fast to encode and decode (in the neighborhood of 200-300MB/s on my core2 desktop, when using the C++ bindings). It's just the Python implementation that is slow. I'm working on a Python implementation that will be much faster. It's really unfortunately that Protocol Buffers are getting a bad rap due to the current Python implementation.