3 ms·

Google's own protobuf runtime is so bloated that even google themselves don't use it in many of their software, but rather nanopb[1], which is done by completel

Google's own protobuf runtime is so bloated that even google themselves don't use it in many of their software, but rather nanopb[1], which is done by completely unrelated person to google. 1: https://jpa.kapsi.fi/nanopb/ https://jpa.kapsi.fi/nanopb/

jeffbee 6y ago
I don’t think that’s true in google3. People often mistake the scale of Google’s internal code with respect to their visible external code.
- jmkjaer 6y ago
  For anyone unfamiliar, Google3 is apparently Google's monorepo, according to this HN comment: https://news.ycombinator.com/item?id=17123620 https://news.ycombinator.com/item?id=17123620
  - alexhutcheson 6y ago
    https://research.google/pubs/pub45424/ https://research.google/pubs/pub45424/
vlovich123 6y ago
I am only aware of nanopb being used at Google in embedded scenarios. The primary reason for the project I worked on was that the first Pixel Buds didn't have a C++ compiler (at least not a functional one - based on GCC 2 or 3 if I recall correctly & an extremely crippled implementation). Pixel Buds 2 didn't have that limitation (stock GCC 9) but nanopb was kept for the second Pixel Buds because the memory management was saner for an embedded RTOS environment (vs stock upstream that might use STL or otherwise try to allocate memory). I don't think runtime bloat was really on our radar (it could have been the next critical path, but wasn't on our radar at the time). Nanopb itself needed some changes to the codegen for the first Pixel Buds (which I investigated & fixed) due to architectural peculiarities of the CSR8675. It generated a lot of constants but the CSR8675 had limited space there & was made worse because every 8 bits took up 1 16-bit word (e.g. the size of your string literal or any constant byte array is effectively doubled). So I changed the code gen to put those constants in the code section instead. Before this change resolved the issue more fully, random engineers who were unlucky enough to add a constant in some way were hitting limits internally with regularity (as the project was heading toward ship) & working around it by removing characters from log messages (e.g. "Some long message" => "Sme lng msg"). For what it's worth I tried to contribute any meaningful improvements we made back to nanopb but this one didn't make that cut as it was specific to a chip no one in the wider community would be using anyway (8675 is super old). I don't exclude the possibility of it using in more places, but I'd say the average Google engineer does not encounter nanopb in their daily development.
brandmeyer 6y ago
> even google themselves don't use it in many of their software Citation needed. Granted its a big company, so its certainly possible that some teams I'm unaware of were using it. But I never ran into one. Besides, nanopb doesn't answer the critique of the article. Nanopb cuts down on heap traffic by statically allocating flat buffers large enough to hold maximums set by the application. It solves the two-pass problem by the simple expediency of actually running two passes, storing the size info on the ordinary call stack. In effect, it trades malloc/free traffic for higher peak memory utilization. At the largest scales, big G suffers from total memory pressure more than it suffers from lack of arithmetic. Nanopb isn't a good trade for them. It is a good trade for my current (hard realtime, embedded) application.
- Cloudef 6y ago
  They use it in android. They also use it in their firebase sdks for ios/android. Nanopb doesnt solve wire issues. Thats for you to solve by designing your data better. Nanopb solves your applications binary size getting needlessly huge, due to less bloated code-gen and saner runtime (if you have to use protobuf, or are already married with it that is). It also gives more control over memory management, which is especially important for embedded.
googlyengineer 6y ago
As a google engineer, I would appreciate it if you did not spurt nonsense. Unless you have some clear evidence of your claims.

jeffbee 6y ago

I don’t think that’s true in google3. People often mistake the scale of Google’s internal code with respect to their visible external code.

jmkjaer 6y ago
For anyone unfamiliar, Google3 is apparently Google's monorepo, according to this HN comment: https://news.ycombinator.com/item?id=17123620 https://news.ycombinator.com/item?id=17123620
- alexhutcheson 6y ago
  https://research.google/pubs/pub45424/ https://research.google/pubs/pub45424/

vlovich123 6y ago

I am only aware of nanopb being used at Google in embedded scenarios. The primary reason for the project I worked on was that the first Pixel Buds didn't have a C++ compiler (at least not a functional one - based on GCC 2 or 3 if I recall correctly & an extremely crippled implementation). Pixel Buds 2 didn't have that limitation (stock GCC 9) but nanopb was kept for the second Pixel Buds because the memory management was saner for an embedded RTOS environment (vs stock upstream that might use STL or otherwise try to allocate memory). I don't think runtime bloat was really on our radar (it could have been the next critical path, but wasn't on our radar at the time). Nanopb itself needed some changes to the codegen for the first Pixel Buds (which I investigated & fixed) due to architectural peculiarities of the CSR8675. It generated a lot of constants but the CSR8675 had limited space there & was made worse because every 8 bits took up 1 16-bit word (e.g. the size of your string literal or any constant byte array is effectively doubled). So I changed the code gen to put those constants in the code section instead. Before this change resolved the issue more fully, random engineers who were unlucky enough to add a constant in some way were hitting limits internally with regularity (as the project was heading toward ship) & working around it by removing characters from log messages (e.g. "Some long message" => "Sme lng msg"). For what it's worth I tried to contribute any meaningful improvements we made back to nanopb but this one didn't make that cut as it was specific to a chip no one in the wider community would be using anyway (8675 is super old). I don't exclude the possibility of it using in more places, but I'd say the average Google engineer does not encounter nanopb in their daily development.

brandmeyer 6y ago

> even google themselves don't use it in many of their software Citation needed. Granted its a big company, so its certainly possible that some teams I'm unaware of were using it. But I never ran into one. Besides, nanopb doesn't answer the critique of the article. Nanopb cuts down on heap traffic by statically allocating flat buffers large enough to hold maximums set by the application. It solves the two-pass problem by the simple expediency of actually running two passes, storing the size info on the ordinary call stack. In effect, it trades malloc/free traffic for higher peak memory utilization. At the largest scales, big G suffers from total memory pressure more than it suffers from lack of arithmetic. Nanopb isn't a good trade for them. It is a good trade for my current (hard realtime, embedded) application.

Cloudef 6y ago
They use it in android. They also use it in their firebase sdks for ios/android. Nanopb doesnt solve wire issues. Thats for you to solve by designing your data better. Nanopb solves your applications binary size getting needlessly huge, due to less bloated code-gen and saner runtime (if you have to use protobuf, or are already married with it that is). It also gives more control over memory management, which is especially important for embedded.

googlyengineer 6y ago

As a google engineer, I would appreciate it if you did not spurt nonsense. Unless you have some clear evidence of your claims.