9 ms·

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C

by tijsvd 6y ago

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at least some implementation available. Then, the author laments the wire format itself for having varint-encoded length prefixes, which can not be fixed up. That is true, but it's not that much of a problem. Most straightforward is to simply go through nested data multiple times, once to calculate the length, and again for the actual encoding (and then again and again for deeper nesting). What makes this bearable is the fact that data will be mostly loaded into L2 cache (L1 for smaller messages) on the first pass, which makes the next pass much faster. The story breaks down for large, deeply nested messages, but then, the topic here is telemetry which I would expect to consist of a stream of small, shallow messages.

thisiscorrect 6y ago
Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated? [edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?
- gabereiser 6y ago
  Probably a complaint of the codegen and it’s wacky name mangling.
- notretarded 6y ago
  Probably all this "security" nonsense. What could possibly go wrong with implementing a binary serialisation protocol your self...
  - inetknght 6y ago
    Oh look sarcasm! On Hacker News!
- tijsvd 6y ago
  Things may have improved since, but the implementations are somehow very large and slow. Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages, and perhaps even for optional scalars. This may be optimal for larger documents, but not for smallish messages (my use case was market data and trading instructions). I measured certain small messages, where an encode/decode pair would take over a microsecond with Google's implementation, but about 50 ns with a simpler one (versus 15 ns for a memcpy). For Java my experience is mostly with the API itself, which felt very heavy. Edit: I think a lot depends on your use case. I use protobuf mostly as a 'trusted' protocol. If someone didn't set a required field, I don't care. Some bloat may have to do with verifications that I've never needed.
  - haberman 6y ago
    > Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages This is no longer the case if you use arenas: https://developers.google.com/protocol-buffers/docs/reference/arenas https://developers.google.com/protocol-buffers/docs/referenc... > and perhaps even for optional scalars This has never been the case, except for string fields where std::string forces us to allocate. Ideally we will eventually use std::string_view for string accessors instead of std::string, so that even string data can be allocated on an arena instead of the heap.
    - seppel 6y ago
      > This has never been the case, except for string fields where std::string forces us to allocate. I'm was quite surprised you didnt offer your own stringview implementation (or something similar) the last time I looked at protobuf. I'd naively assume that inside Google this could be quite a low-efford high-reward optimization.
      - haberman 6y ago
        > I'm was quite surprised you didnt offer your own stringview implementation (or something similar) the last time I looked at protobuf. We sort of do actually: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/stubs/stringpiece.h https://github.com/protocolbuffers/protobuf/blob/master/src/... The internal version of protobuf lets you switch individual string fields to string_view using [ctype=STRING_PIECE], but migrating the default away from std::string is mainly just an enormous migration challenge. Internally we also do something slightly nuts: we break the encapsulation of std::string so that we can point it to arena-allocated memory (we then "steal" the memory back before the destructor runs). We can only afford to do this internally, where the implementation of std::string is known. The real long-term solution is to move to string_view.
haberman 6y ago
The author of this piece is wanting to stream possibly multi-MB messages with sub-linear memory use. See more in our Twitter conversation here: https://twitter.com/richardstartin/status/1344068132978221056 https://twitter.com/richardstartin/status/134406813297822105...
- Animats 6y ago
  "possibly multi-MB messages" Megabytes? By "telemetry" does he mean "getting info on what our pumps out in the oil field are doing" or "snooping on users of a program"?
  - deleted 6y ago
    [deleted]
  - brown9-2 6y ago
    think of the type of monitoring/observability data that Datadog's agent collects
  - viraptor 6y ago
    People tend to jump from telemetry straight to snooping on users even though it's just one possible case. A possible megabyte of data: A rolling in-memory debug-level log + stack traces with context + timing information, that you snapshot when the app catches and exception. Or a buffered set of timing stats from a minute of runtime. The author mentioned implementing ddsketch for probabilistic quantile distribution. That's something you run on your own app/infrastructure rather than for snooping on users.
  - sebastos 6y ago
    This over-assuming-of-shared-context is one of my pet peeves on this site. Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow. Of course, I'm also annoyed that some computer science PhD apparently googled around for a cooler name for multi-dimensional arrays and ruined the term "Tensor" forevermore.
TickleSteve 6y ago
pb is also used in low-resource/embedded systems (to send the data to the servers). In these cases, the processors typically have little to no cache and very limited RAM, Performing multiple encoding passes is the only option and each pass is as expensive as the first.
- tijsvd 6y ago
  In these environments you'd typically take special care to keep messages flat, I think - in which case there's no problem with multiple passes.
haberman 6y ago
> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). As someone who has been working on protobuf-related things for >10 years, including creating a size-focused implementation (https://github.com/protocolbuffers/upb https://github.com/protocolbuffers/upb), and has been working on the protobuf team for >5 years, I have a few thoughts on this (thoughts are my own, and I don't speak for anybody else). I think it is true that protobuf C++ could be a lot more lean than it currently is (I can't speak to Java as I don't work on it directly). That's why I created upb to begin with. But there's also a bit more to this story. The protobuf core runtime is split into two parts, "lite" and "full". If you don't need reflection for your protos, it's better to use "lite" by using "option optimize_for = LITE_RUNTIME" in your .proto file (https://developers.google.com/protocol-buffers/docs/proto#options https://developers.google.com/protocol-buffers/docs/proto#op...). That will cut out a huge amount of code size from your binary. On the downside, you won't get functionality that requires reflection, such as text format, JSON, and DebugString(). Even the lite runtime can get "lighter" if you compile your binary to statically link the runtime and strip unused symbols with -ffunction-sections/-fdata-sections/--gc-sections flags. Some parts of the lite runtime are only needed in unusual situations, like ExtensionSet which is only used if your protos use proto2 extensions (https://developers.google.com/protocol-buffers/docs/proto#extensions https://developers.google.com/protocol-buffers/docs/proto#ex...). If you avoid these cases, the lite runtime is quite light. However, there is also the issue of the generated code size. Unlike the runtime, this is not a fixed cost, but is proportional to the number of messages you use. If you have a lot of messages it can quickly dwarf the size of the runtime. For this reason, C++ also supports "option optimize_for = CODE_SIZE" which uses reflection-based algorithms for all parsing/serialization/etc instead of using generated code. This means you pay the fixed size hit from linking in the full runtime, but the generated code size is much smaller. On the downside, "optimize_for = CODE_SIZE" has a severe ~10x speed penalty for parsing and serialization. I have long had the goal of making https://github.com/protocolbuffers/upb https://github.com/protocolbuffers/upb competitive with protobuf C++ in speed while achieving much smaller code size. With the benefit of 10 years of hindsight and many wrong turns, upb is beginning to meet and even surpass these goals. It is an order of magnitude smaller than protobuf C++, both in the core runtime and the generated code, and after some recent experiments it is beginning to significantly surpass it in speed also (I want to publish these results soon, but the code was merged in this PR: https://github.com/protocolbuffers/upb/pull/310 https://github.com/protocolbuffers/upb/pull/310). upb has downsides that prevent it from being fully "user ready" yet: the API is still not 100% stable, there is no C++ API for the generated code yet (and C APIs for protobuf are relatively verbose and painful), it has a bunch of legacy APIs sitting around that I am just on the verge of being able to finally delete, and it doesn't support proto2 extensions yet. On the upside, upb is 100% conformant on every other protobuf feature, it supports reflection, JSON, and text format, but also lets you omit these if you don't want to pay the code size. I hope 2021 is a year when I'll be able to publish more about these results, and when upb will be a more viable choice for users who want a smaller protobuf implementation.
kentonv 6y ago
> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Right, they were designed for use in Google's servers, where binary size is mostly irrelevant, while speed and features (e.g. reflection) are valued. "Lite mode" (not mentioned in the article, for some reason) optimizes for code size instead. Admittedly, it's not as small as an implementation written from scratch with minimal code footprint as the main goal, but it does cut the size quite a bit... (Disclaimer: I wrote those C++ and Java implementations, but that was 10-15 years ago, things may have changed in the meantime...)
- girvo 6y ago
  HN is one of the few places you can talk smack about a companies tooling and have the person who wrote it reply! Lite mode is neat, I hadn’t seen that. That’s super useful for my use case, cheers. Any gotchas with it to be aware of?
  - kentonv 6y ago
    > Any gotchas with it to be aware of? Lots, unfortunately. E.g. you can't stringify objects for debug printing, because stringification is implemented in terms of reflection.
  - tailspin2019 6y ago
    > HN is one of the few places you can talk smack about a companies tooling and have the person who wrote it reply! Not just that, I suspect it’s a sign of HN’s unique culture that the “smack talk” seems not to be taken personally and instead a highly nuanced and interesting discussion ensues!
keyle 6y ago
If you truly wanted speed, why not Cap'n'Proto?
- tijsvd 6y ago
  It didn't exist yet. We were evolving from simple raw messages with all sorts of problems. The alternative was something custom again, with better support for schema evolution, but pb was convenient due to existing implementations in Python (system tests) and C# (UI).
saberience 6y ago
You don't work for Blizzard do you? When I worked there engineers (not my team thankfully) designed their own alternative to protobuf to try and save more bytes over the wire, which in my opinion was a really poor decision. Rather than get on with actually adding value, they ended up pushing back multiple other teams deadlines while adding almost no value. It was a classic "not built here" mentality and doing engineering work for the sake of the work, rather than actually ask "what problem are we trying to solve here?" If protobuf works for Google then it essentially works for 99.999% of every other company on the globe.
- rkagerer 6y ago
  If protobuf works for Google then it essentially works for 99.999% of every other company on the globe. I can't agree with this mindset. Another commenter here pointed out the Google implementation is 20x slower than others, and 1.6MB for this kind of task feels bloaty. Just because it meets Google's needs doesn't mean it's universally adequate.
  - ikiris 6y ago
    Why exactly does a megabyte matter, unless you're trying to run on a 90's watch or embedded?
    - FpUser 6y ago
      Maybe because "20X slower"
      - dweis 6y ago
        These are simply tradeoffs. For example, one can make very fast protobuf parsers/serializers if you prioritize that over the ergonomics of the generated code.
    - naniwaduni 6y ago
      Well, transmitting a megabyte also takes a full second at 8 Mbps.
Scaevolus 6y ago
Here's a minimal Python protobuf parser that can extract basic information in about 50 LOC. https://github.com/kubernetes/test-infra/blob/master/gubernator/pb_glance.py https://github.com/kubernetes/test-infra/blob/master/guberna...
cbsmith 6y ago
Yeah, the author seems to both understand the nesting issue, and yet not understand how it's not an issue. As per the format guidelines, you shouldn't have deeply nested structures... or at least if you do it should be in cases where only one or two layers of nesting are decoded at any given stage (I've done this with "envelope" messages). Of course, there are ways to write serialized lengths of nested elements recursively, you just have to do it yourself. It's just not what you normally want to do, so the standard library and compiler don't emit that logic. If you are going down that road, you gotta ask yourself why.
marco_craveiro 6y ago
Its a shame you can't share the code, we could really do with a modern C++ implementation. Having said that, protozero looks very interesting [1], need to find some time to look into it. [1] https://github.com/mapbox/protozero https://github.com/mapbox/protozero

by tijsvd 6y ago

thisiscorrect 6y ago

Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated? [edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?

gabereiser 6y ago
Probably a complaint of the codegen and it’s wacky name mangling.
notretarded 6y ago
Probably all this "security" nonsense. What could possibly go wrong with implementing a binary serialisation protocol your self...
- inetknght 6y ago
  Oh look sarcasm! On Hacker News!
tijsvd 6y ago
Things may have improved since, but the implementations are somehow very large and slow. Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages, and perhaps even for optional scalars. This may be optimal for larger documents, but not for smallish messages (my use case was market data and trading instructions). I measured certain small messages, where an encode/decode pair would take over a microsecond with Google's implementation, but about 50 ns with a simpler one (versus 15 ns for a memcpy). For Java my experience is mostly with the API itself, which felt very heavy. Edit: I think a lot depends on your use case. I use protobuf mostly as a 'trusted' protocol. If someone didn't set a required field, I don't care. Some bloat may have to do with verifications that I've never needed.
- haberman 6y ago
  > Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages This is no longer the case if you use arenas: https://developers.google.com/protocol-buffers/docs/reference/arenas https://developers.google.com/protocol-buffers/docs/referenc... > and perhaps even for optional scalars This has never been the case, except for string fields where std::string forces us to allocate. Ideally we will eventually use std::string_view for string accessors instead of std::string, so that even string data can be allocated on an arena instead of the heap.
  - seppel 6y ago
    > This has never been the case, except for string fields where std::string forces us to allocate. I'm was quite surprised you didnt offer your own stringview implementation (or something similar) the last time I looked at protobuf. I'd naively assume that inside Google this could be quite a low-efford high-reward optimization.
    - haberman 6y ago
      > I'm was quite surprised you didnt offer your own stringview implementation (or something similar) the last time I looked at protobuf. We sort of do actually: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/stubs/stringpiece.h https://github.com/protocolbuffers/protobuf/blob/master/src/... The internal version of protobuf lets you switch individual string fields to string_view using [ctype=STRING_PIECE], but migrating the default away from std::string is mainly just an enormous migration challenge. Internally we also do something slightly nuts: we break the encapsulation of std::string so that we can point it to arena-allocated memory (we then "steal" the memory back before the destructor runs). We can only afford to do this internally, where the implementation of std::string is known. The real long-term solution is to move to string_view.

haberman 6y ago

The author of this piece is wanting to stream possibly multi-MB messages with sub-linear memory use. See more in our Twitter conversation here: https://twitter.com/richardstartin/status/1344068132978221056 https://twitter.com/richardstartin/status/134406813297822105...

Animats 6y ago
"possibly multi-MB messages" Megabytes? By "telemetry" does he mean "getting info on what our pumps out in the oil field are doing" or "snooping on users of a program"?
- deleted 6y ago
  [deleted]
- brown9-2 6y ago
  think of the type of monitoring/observability data that Datadog's agent collects
- viraptor 6y ago
  People tend to jump from telemetry straight to snooping on users even though it's just one possible case. A possible megabyte of data: A rolling in-memory debug-level log + stack traces with context + timing information, that you snapshot when the app catches and exception. Or a buffered set of timing stats from a minute of runtime. The author mentioned implementing ddsketch for probabilistic quantile distribution. That's something you run on your own app/infrastructure rather than for snooping on users.
- sebastos 6y ago
  This over-assuming-of-shared-context is one of my pet peeves on this site. Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow. Of course, I'm also annoyed that some computer science PhD apparently googled around for a cooler name for multi-dimensional arrays and ruined the term "Tensor" forevermore.

TickleSteve 6y ago

pb is also used in low-resource/embedded systems (to send the data to the servers). In these cases, the processors typically have little to no cache and very limited RAM, Performing multiple encoding passes is the only option and each pass is as expensive as the first.

tijsvd 6y ago
In these environments you'd typically take special care to keep messages flat, I think - in which case there's no problem with multiple passes.

haberman 6y ago

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). As someone who has been working on protobuf-related things for >10 years, including creating a size-focused implementation (https://github.com/protocolbuffers/upb https://github.com/protocolbuffers/upb), and has been working on the protobuf team for >5 years, I have a few thoughts on this (thoughts are my own, and I don't speak for anybody else). I think it is true that protobuf C++ could be a lot more lean than it currently is (I can't speak to Java as I don't work on it directly). That's why I created upb to begin with. But there's also a bit more to this story. The protobuf core runtime is split into two parts, "lite" and "full". If you don't need reflection for your protos, it's better to use "lite" by using "option optimize_for = LITE_RUNTIME" in your .proto file (https://developers.google.com/protocol-buffers/docs/proto#options https://developers.google.com/protocol-buffers/docs/proto#op...). That will cut out a huge amount of code size from your binary. On the downside, you won't get functionality that requires reflection, such as text format, JSON, and DebugString(). Even the lite runtime can get "lighter" if you compile your binary to statically link the runtime and strip unused symbols with -ffunction-sections/-fdata-sections/--gc-sections flags. Some parts of the lite runtime are only needed in unusual situations, like ExtensionSet which is only used if your protos use proto2 extensions (https://developers.google.com/protocol-buffers/docs/proto#extensions https://developers.google.com/protocol-buffers/docs/proto#ex...). If you avoid these cases, the lite runtime is quite light. However, there is also the issue of the generated code size. Unlike the runtime, this is not a fixed cost, but is proportional to the number of messages you use. If you have a lot of messages it can quickly dwarf the size of the runtime. For this reason, C++ also supports "option optimize_for = CODE_SIZE" which uses reflection-based algorithms for all parsing/serialization/etc instead of using generated code. This means you pay the fixed size hit from linking in the full runtime, but the generated code size is much smaller. On the downside, "optimize_for = CODE_SIZE" has a severe ~10x speed penalty for parsing and serialization. I have long had the goal of making https://github.com/protocolbuffers/upb https://github.com/protocolbuffers/upb competitive with protobuf C++ in speed while achieving much smaller code size. With the benefit of 10 years of hindsight and many wrong turns, upb is beginning to meet and even surpass these goals. It is an order of magnitude smaller than protobuf C++, both in the core runtime and the generated code, and after some recent experiments it is beginning to significantly surpass it in speed also (I want to publish these results soon, but the code was merged in this PR: https://github.com/protocolbuffers/upb/pull/310 https://github.com/protocolbuffers/upb/pull/310). upb has downsides that prevent it from being fully "user ready" yet: the API is still not 100% stable, there is no C++ API for the generated code yet (and C APIs for protobuf are relatively verbose and painful), it has a bunch of legacy APIs sitting around that I am just on the verge of being able to finally delete, and it doesn't support proto2 extensions yet. On the upside, upb is 100% conformant on every other protobuf feature, it supports reflection, JSON, and text format, but also lets you omit these if you don't want to pay the code size. I hope 2021 is a year when I'll be able to publish more about these results, and when upb will be a more viable choice for users who want a smaller protobuf implementation.

kentonv 6y ago

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Right, they were designed for use in Google's servers, where binary size is mostly irrelevant, while speed and features (e.g. reflection) are valued. "Lite mode" (not mentioned in the article, for some reason) optimizes for code size instead. Admittedly, it's not as small as an implementation written from scratch with minimal code footprint as the main goal, but it does cut the size quite a bit... (Disclaimer: I wrote those C++ and Java implementations, but that was 10-15 years ago, things may have changed in the meantime...)

girvo 6y ago
HN is one of the few places you can talk smack about a companies tooling and have the person who wrote it reply! Lite mode is neat, I hadn’t seen that. That’s super useful for my use case, cheers. Any gotchas with it to be aware of?
- kentonv 6y ago
  > Any gotchas with it to be aware of? Lots, unfortunately. E.g. you can't stringify objects for debug printing, because stringification is implemented in terms of reflection.
- tailspin2019 6y ago
  > HN is one of the few places you can talk smack about a companies tooling and have the person who wrote it reply! Not just that, I suspect it’s a sign of HN’s unique culture that the “smack talk” seems not to be taken personally and instead a highly nuanced and interesting discussion ensues!

keyle 6y ago

If you truly wanted speed, why not Cap'n'Proto?

tijsvd 6y ago
It didn't exist yet. We were evolving from simple raw messages with all sorts of problems. The alternative was something custom again, with better support for schema evolution, but pb was convenient due to existing implementations in Python (system tests) and C# (UI).

saberience 6y ago

You don't work for Blizzard do you? When I worked there engineers (not my team thankfully) designed their own alternative to protobuf to try and save more bytes over the wire, which in my opinion was a really poor decision. Rather than get on with actually adding value, they ended up pushing back multiple other teams deadlines while adding almost no value. It was a classic "not built here" mentality and doing engineering work for the sake of the work, rather than actually ask "what problem are we trying to solve here?" If protobuf works for Google then it essentially works for 99.999% of every other company on the globe.

rkagerer 6y ago
If protobuf works for Google then it essentially works for 99.999% of every other company on the globe. I can't agree with this mindset. Another commenter here pointed out the Google implementation is 20x slower than others, and 1.6MB for this kind of task feels bloaty. Just because it meets Google's needs doesn't mean it's universally adequate.
- ikiris 6y ago
  Why exactly does a megabyte matter, unless you're trying to run on a 90's watch or embedded?
  - FpUser 6y ago
    Maybe because "20X slower"
    - dweis 6y ago
      These are simply tradeoffs. For example, one can make very fast protobuf parsers/serializers if you prioritize that over the ergonomics of the generated code.
  - naniwaduni 6y ago
    Well, transmitting a megabyte also takes a full second at 8 Mbps.

Scaevolus 6y ago

Here's a minimal Python protobuf parser that can extract basic information in about 50 LOC. https://github.com/kubernetes/test-infra/blob/master/gubernator/pb_glance.py https://github.com/kubernetes/test-infra/blob/master/guberna...

cbsmith 6y ago

Yeah, the author seems to both understand the nesting issue, and yet not understand how it's not an issue. As per the format guidelines, you shouldn't have deeply nested structures... or at least if you do it should be in cases where only one or two layers of nesting are decoded at any given stage (I've done this with "envelope" messages). Of course, there are ways to write serialized lengths of nested elements recursively, you just have to do it yourself. It's just not what you normally want to do, so the standard library and compiler don't emit that logic. If you are going down that road, you gotta ask yourself why.

marco_craveiro 6y ago

Its a shame you can't share the code, we could really do with a modern C++ implementation. Having said that, protozero looks very interesting [1], need to find some time to look into it. [1] https://github.com/mapbox/protozero https://github.com/mapbox/protozero