4 ms·

His point on protobuf-java library adding nontrivial bloat to his Java app is definitely valid. However, his other argument about protobuf wire format being ine

by xyzzyz 6y ago

His point on protobuf-java library adding nontrivial bloat to his Java app is definitely valid. However, his other argument about protobuf wire format being inefficient is hard to square with decades of practical experience Google had with protobufs, which are used for literally everything, including telemetry, and high throughput, low latency applications. Sure, having to recursively precompute lengths before serialization is a bit of a hassle, but I wouldn’t call it expensive.

sillysaurusx 6y ago
Signal boosting this. I am the last person to be swayed by “use our magical design that solves all problems,” but in the case of protobufs, it solves most of them. I didn’t like it until I was forced to use it. Now I’m not sure I’ll ever go back. There’s a handy code snippet to turn any protobuf message into JSON: from google.protobuf.json_format import MessageToJson import json from pprint import pprint as pp pp(json.loads(MessageToJson(msg))) Examples of turning arbitrary tensorflow objects into JSON: https://twitter.com/theshawwn/status/1331625739993575427?s=21 https://twitter.com/theshawwn/status/1331625739993575427?s=2...
- intricatedetail 6y ago
  Why they use Java like code in Python? "MessageToJson" doesn't look idiomatic.
  - singron 6y ago
    CapWords function names matches the C++ style guide at Google, not java. A lot of python uses autogenerated bindings to C++ code and reuses the same naming conventions. Even pure python code would use that style although their public styleguide doesn't mention it anymore.
  - ipsum2 6y ago
    The official protobuf library for Python is not pythonic at all.
  - q3k 6y ago
    This is because this follows Google’s internal Python styeguide, which was established well before PEP8 and even Python 2.x.
    - joshuamorton 6y ago
      Notably this is no longer true[0]. While legacy stuff can still use the CapWords styling, snake_case names are preferred for all new methods and functions (including code that wraps c++). But this is a relatively recent change to the style guide, and the proto libraries far predate it. (As a response to a sibling of the parent, the same rules apply to internal code, CapWords should only be used for consistency within a file. All new code should be embrace it's snakey heritage). [0]: https://google.github.io/styleguide/pyguide.html#3162-naming-conventions https://google.github.io/styleguide/pyguide.html#3162-naming...
WorldMaker 6y ago
It seems easy to square the practical experience of protobufs at Google versus the tradeoffs encountered in the wire format when you take into account how many Google engineers have directly contributed to the diaspora of related formats such as CapnProto, flatbuffers, msgpack, etc with different tradeoffs (especially for smaller scales than Google). It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. That's a valid tradeoff for Google scale, that comes at a cost of other potential applications (such as telemetry that can be very time/cost-sensitive at write-time but has much more relaxed read time/cost-sensitivity). > Sure, having to recursively precompute lengths before serialization is a bit of a hassle, but I wouldn’t call it expensive. It's expensive in memory (needing to prebuild all the subcomponents of the message, instead of streaming them as needed), and potentially time (prebuilding then sending, versus streaming as soon as data is available to write). Of course, in the scheme of things it may not be expensive to particular use cases. (Which seems why the author seems to particularly highlight this specific use case where such things really are expensive overhead necessary to avoid.)
- kentonv 6y ago
  > It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. They just did something that worked well enough, and it worked well enough that it was rapidly adopted by the rest of the company. It was then too late to change anything. Yes, the fact that a variable-width size must be written before the data is a kind of big problem, which essentially requires you to make two passes over the message tree, one to compute sizes and one to write the data. There are some clever optimizations that can reduce the impact, but I don't think the designers would make the same decision if starting from scratch with no need to support legacy. But there was no point in history where it was worth breaking compatibility to fix this issue, so that's how it remains. It's a problem, just not that big a problem. (I maintained Protobuf for several years, including writing version 2 and open sourcing it.)

by xyzzyz 6y ago

sillysaurusx 6y ago

Signal boosting this. I am the last person to be swayed by “use our magical design that solves all problems,” but in the case of protobufs, it solves most of them. I didn’t like it until I was forced to use it. Now I’m not sure I’ll ever go back. There’s a handy code snippet to turn any protobuf message into JSON: from google.protobuf.json_format import MessageToJson import json from pprint import pprint as pp pp(json.loads(MessageToJson(msg))) Examples of turning arbitrary tensorflow objects into JSON: https://twitter.com/theshawwn/status/1331625739993575427?s=21 https://twitter.com/theshawwn/status/1331625739993575427?s=2...

intricatedetail 6y ago
Why they use Java like code in Python? "MessageToJson" doesn't look idiomatic.
- singron 6y ago
  CapWords function names matches the C++ style guide at Google, not java. A lot of python uses autogenerated bindings to C++ code and reuses the same naming conventions. Even pure python code would use that style although their public styleguide doesn't mention it anymore.
- ipsum2 6y ago
  The official protobuf library for Python is not pythonic at all.
- q3k 6y ago
  This is because this follows Google’s internal Python styeguide, which was established well before PEP8 and even Python 2.x.
  - joshuamorton 6y ago
    Notably this is no longer true[0]. While legacy stuff can still use the CapWords styling, snake_case names are preferred for all new methods and functions (including code that wraps c++). But this is a relatively recent change to the style guide, and the proto libraries far predate it. (As a response to a sibling of the parent, the same rules apply to internal code, CapWords should only be used for consistency within a file. All new code should be embrace it's snakey heritage). [0]: https://google.github.io/styleguide/pyguide.html#3162-naming-conventions https://google.github.io/styleguide/pyguide.html#3162-naming...

WorldMaker 6y ago

It seems easy to square the practical experience of protobufs at Google versus the tradeoffs encountered in the wire format when you take into account how many Google engineers have directly contributed to the diaspora of related formats such as CapnProto, flatbuffers, msgpack, etc with different tradeoffs (especially for smaller scales than Google). It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. That's a valid tradeoff for Google scale, that comes at a cost of other potential applications (such as telemetry that can be very time/cost-sensitive at write-time but has much more relaxed read time/cost-sensitivity). > Sure, having to recursively precompute lengths before serialization is a bit of a hassle, but I wouldn’t call it expensive. It's expensive in memory (needing to prebuild all the subcomponents of the message, instead of streaming them as needed), and potentially time (prebuilding then sending, versus streaming as soon as data is available to write). Of course, in the scheme of things it may not be expensive to particular use cases. (Which seems why the author seems to particularly highlight this specific use case where such things really are expensive overhead necessary to avoid.)

kentonv 6y ago
> It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. They just did something that worked well enough, and it worked well enough that it was rapidly adopted by the rest of the company. It was then too late to change anything. Yes, the fact that a variable-width size must be written before the data is a kind of big problem, which essentially requires you to make two passes over the message tree, one to compute sizes and one to write the data. There are some clever optimizations that can reduce the impact, but I don't think the designers would make the same decision if starting from scratch with no need to support legacy. But there was no point in history where it was worth breaking compatibility to fix this issue, so that's how it remains. It's a problem, just not that big a problem. (I maintained Protobuf for several years, including writing version 2 and open sourcing it.)