6 ms·

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered).

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). As someone who has been working on protobuf-related things for >10 years, including creating a size-focused implementation (https://github.com/protocolbuffers/upb https://github.com/protocolbuffers/upb), and has been working on the protobuf team for >5 years, I have a few thoughts on this (thoughts are my own, and I don't speak for anybody else). I think it is true that protobuf C++ could be a lot more lean than it currently is (I can't speak to Java as I don't work on it directly). That's why I created upb to begin with. But there's also a bit more to this story. The protobuf core runtime is split into two parts, "lite" and "full". If you don't need reflection for your protos, it's better to use "lite" by using "option optimize_for = LITE_RUNTIME" in your .proto file (https://developers.google.com/protocol-buffers/docs/proto#options https://developers.google.com/protocol-buffers/docs/proto#op...). That will cut out a huge amount of code size from your binary. On the downside, you won't get functionality that requires reflection, such as text format, JSON, and DebugString(). Even the lite runtime can get "lighter" if you compile your binary to statically link the runtime and strip unused symbols with -ffunction-sections/-fdata-sections/--gc-sections flags. Some parts of the lite runtime are only needed in unusual situations, like ExtensionSet which is only used if your protos use proto2 extensions (https://developers.google.com/protocol-buffers/docs/proto#extensions https://developers.google.com/protocol-buffers/docs/proto#ex...). If you avoid these cases, the lite runtime is quite light. However, there is also the issue of the generated code size. Unlike the runtime, this is not a fixed cost, but is proportional to the number of messages you use. If you have a lot of messages it can quickly dwarf the size of the runtime. For this reason, C++ also supports "option optimize_for = CODE_SIZE" which uses reflection-based algorithms for all parsing/serialization/etc instead of using generated code. This means you pay the fixed size hit from linking in the full runtime, but the generated code size is much smaller. On the downside, "optimize_for = CODE_SIZE" has a severe ~10x speed penalty for parsing and serialization. I have long had the goal of making https://github.com/protocolbuffers/upb https://github.com/protocolbuffers/upb competitive with protobuf C++ in speed while achieving much smaller code size. With the benefit of 10 years of hindsight and many wrong turns, upb is beginning to meet and even surpass these goals. It is an order of magnitude smaller than protobuf C++, both in the core runtime and the generated code, and after some recent experiments it is beginning to significantly surpass it in speed also (I want to publish these results soon, but the code was merged in this PR: https://github.com/protocolbuffers/upb/pull/310 https://github.com/protocolbuffers/upb/pull/310). upb has downsides that prevent it from being fully "user ready" yet: the API is still not 100% stable, there is no C++ API for the generated code yet (and C APIs for protobuf are relatively verbose and painful), it has a bunch of legacy APIs sitting around that I am just on the verge of being able to finally delete, and it doesn't support proto2 extensions yet. On the upside, upb is 100% conformant on every other protobuf feature, it supports reflection, JSON, and text format, but also lets you omit these if you don't want to pay the code size. I hope 2021 is a year when I'll be able to publish more about these results, and when upb will be a more viable choice for users who want a smaller protobuf implementation.

tijsvd 6y ago
What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?
- joshuamorton 6y ago
  [Also a googler]: PB is used for a wide variety of things, its a lingua-franca of data interchange within Google, but web (frontend) still often uses json. So if I want to have some proto in a database and render it on a UI to a user, somewhere in there I'm probably going to be translating the raw proto to either json or json-like (I'll admit I'm not sure how stuff like grpc-web fits in here, if you can just get proto-serialized bytes passed around in some a blob, but there are some pretty low-overhead json proto encodings that exist anyway).
  - tijsvd 6y ago
    Sure that makes some sense. But it's somewhat ugly as well, from an architecture point of view. Because that argument translates to anything else you might want to do with these objects. In past libraries for C++ I've tried to prevent this kind of coupling by adding generated template methods like "walk(f)" where f would be a templated callable, called with a descriptor and data reference for each field. Any kind of pretty printer or SQL statement can be built that way.
- kentonv 6y ago
  People implementing public-facing APIs might not be comfortable forcing all the consumers of that API to use Protobuf, but may want to use Protobuf internally within their service. So, Protobuf lets you convert to/from JSON, so that your service can expose a JSON API while maintaining Protobuf schemas and never dealing with a JSON parser directly. The implementation is based on Protobuf's existing reflection features, so if you are already compiling with that enabled, the JSON implementation does not add much code bloat on top of that.
- haberman 6y ago
  > Don't people use pb so they do not have to deal with JSON? I think that's one reason, but there are others too, like getting generated type-safe accessors and an explicit schema. I've sometimes used protobuf for this reason even when I'm not planning to use binary format. > I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size? Here is a breakdown of a binary that uses upb and links in binary format, JSON format, text format, and some generated protos: VM SIZE --------------- 20.6% 26.5Ki Generated Reflection 45.7% 12.1Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upbdefs.c 39.3% 10.4Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upbdefs.c 5.0% 1.33Ki bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upbdefs.c 3.1% 832 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upbdefs.c 2.4% 648 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upbdefs.c 1.2% 320 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/timestamp.upbdefs.c 1.2% 312 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upbdefs.c 1.1% 304 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/field_mask.upbdefs.c 1.1% 296 [1 Others] 19.1% 24.5Ki JSON 60.4% 14.8Ki upb/json_decode.c 39.6% 9.67Ki upb/json_encode.c 14.7% 18.9Ki Reflection Runtime 81.9% 15.4Ki upb/def.c 18.1% 3.41Ki upb/reflection.c 13.9% 17.8Ki Generated Code 46.9% 8.34Ki bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upb.c 17.6% 3.12Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upb.c 16.4% 2.92Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upb.c 14.7% 2.61Ki bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/descriptor.upb.c 2.0% 356 bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upb.c 1.4% 264 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upb.c 0.5% 88 [2 Others] 0.3% 48 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/any.upb.c 0.3% 48 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upb.c 10.0% 12.9Ki Binary Format 54.0% 6.95Ki upb/decode.c 46.0% 5.92Ki upb/encode.c 8.8% 11.3Ki Core Runtime 66.3% 7.52Ki upb/table.c 17.8% 2.02Ki upb/upb.c 15.9% 1.80Ki upb/msg.c 5.6% 7.19Ki [34 Others] 3.8% 4.94Ki Text Format 100.0% 4.94Ki upb/text_encode.c 3.5% 4.46Ki [section .rodata] 100.0% 128Ki TOTAL So JSON is pulling in 25Ki directly, plus it depends on the 19Ki reflection runtime and the generated reflection. But it is all a la carte: if you don't want JSON, just don't use it and don't link it in. (btw that 7Ki for upb/table.c is to support protobuf maps, I want to make that a la carte also so that you don't pay for it when you don't use maps).

tijsvd 6y ago

What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

joshuamorton 6y ago
[Also a googler]: PB is used for a wide variety of things, its a lingua-franca of data interchange within Google, but web (frontend) still often uses json. So if I want to have some proto in a database and render it on a UI to a user, somewhere in there I'm probably going to be translating the raw proto to either json or json-like (I'll admit I'm not sure how stuff like grpc-web fits in here, if you can just get proto-serialized bytes passed around in some a blob, but there are some pretty low-overhead json proto encodings that exist anyway).
- tijsvd 6y ago
  Sure that makes some sense. But it's somewhat ugly as well, from an architecture point of view. Because that argument translates to anything else you might want to do with these objects. In past libraries for C++ I've tried to prevent this kind of coupling by adding generated template methods like "walk(f)" where f would be a templated callable, called with a descriptor and data reference for each field. Any kind of pretty printer or SQL statement can be built that way.
kentonv 6y ago
People implementing public-facing APIs might not be comfortable forcing all the consumers of that API to use Protobuf, but may want to use Protobuf internally within their service. So, Protobuf lets you convert to/from JSON, so that your service can expose a JSON API while maintaining Protobuf schemas and never dealing with a JSON parser directly. The implementation is based on Protobuf's existing reflection features, so if you are already compiling with that enabled, the JSON implementation does not add much code bloat on top of that.
haberman 6y ago
> Don't people use pb so they do not have to deal with JSON? I think that's one reason, but there are others too, like getting generated type-safe accessors and an explicit schema. I've sometimes used protobuf for this reason even when I'm not planning to use binary format. > I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size? Here is a breakdown of a binary that uses upb and links in binary format, JSON format, text format, and some generated protos: VM SIZE --------------- 20.6% 26.5Ki Generated Reflection 45.7% 12.1Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upbdefs.c 39.3% 10.4Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upbdefs.c 5.0% 1.33Ki bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upbdefs.c 3.1% 832 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upbdefs.c 2.4% 648 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upbdefs.c 1.2% 320 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/timestamp.upbdefs.c 1.2% 312 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upbdefs.c 1.1% 304 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/field_mask.upbdefs.c 1.1% 296 [1 Others] 19.1% 24.5Ki JSON 60.4% 14.8Ki upb/json_decode.c 39.6% 9.67Ki upb/json_encode.c 14.7% 18.9Ki Reflection Runtime 81.9% 15.4Ki upb/def.c 18.1% 3.41Ki upb/reflection.c 13.9% 17.8Ki Generated Code 46.9% 8.34Ki bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upb.c 17.6% 3.12Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upb.c 16.4% 2.92Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upb.c 14.7% 2.61Ki bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/descriptor.upb.c 2.0% 356 bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upb.c 1.4% 264 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upb.c 0.5% 88 [2 Others] 0.3% 48 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/any.upb.c 0.3% 48 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upb.c 10.0% 12.9Ki Binary Format 54.0% 6.95Ki upb/decode.c 46.0% 5.92Ki upb/encode.c 8.8% 11.3Ki Core Runtime 66.3% 7.52Ki upb/table.c 17.8% 2.02Ki upb/upb.c 15.9% 1.80Ki upb/msg.c 5.6% 7.19Ki [34 Others] 3.8% 4.94Ki Text Format 100.0% 4.94Ki upb/text_encode.c 3.5% 4.46Ki [section .rodata] 100.0% 128Ki TOTAL So JSON is pulling in 25Ki directly, plus it depends on the 19Ki reflection runtime and the generated reflection. But it is all a la carte: if you don't want JSON, just don't use it and don't link it in. (btw that 7Ki for upb/table.c is to support protobuf maps, I want to make that a la carte also so that you don't pay for it when you don't use maps).