3 ms·

What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON

by tijsvd 6y ago

What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

joshuamorton 6y ago
[Also a googler]: PB is used for a wide variety of things, its a lingua-franca of data interchange within Google, but web (frontend) still often uses json. So if I want to have some proto in a database and render it on a UI to a user, somewhere in there I'm probably going to be translating the raw proto to either json or json-like (I'll admit I'm not sure how stuff like grpc-web fits in here, if you can just get proto-serialized bytes passed around in some a blob, but there are some pretty low-overhead json proto encodings that exist anyway).
- tijsvd 6y ago
  Sure that makes some sense. But it's somewhat ugly as well, from an architecture point of view. Because that argument translates to anything else you might want to do with these objects. In past libraries for C++ I've tried to prevent this kind of coupling by adding generated template methods like "walk(f)" where f would be a templated callable, called with a descriptor and data reference for each field. Any kind of pretty printer or SQL statement can be built that way.
kentonv 6y ago
People implementing public-facing APIs might not be comfortable forcing all the consumers of that API to use Protobuf, but may want to use Protobuf internally within their service. So, Protobuf lets you convert to/from JSON, so that your service can expose a JSON API while maintaining Protobuf schemas and never dealing with a JSON parser directly. The implementation is based on Protobuf's existing reflection features, so if you are already compiling with that enabled, the JSON implementation does not add much code bloat on top of that.
haberman 6y ago
> Don't people use pb so they do not have to deal with JSON? I think that's one reason, but there are others too, like getting generated type-safe accessors and an explicit schema. I've sometimes used protobuf for this reason even when I'm not planning to use binary format. > I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size? Here is a breakdown of a binary that uses upb and links in binary format, JSON format, text format, and some generated protos: VM SIZE --------------- 20.6% 26.5Ki Generated Reflection 45.7% 12.1Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upbdefs.c 39.3% 10.4Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upbdefs.c 5.0% 1.33Ki bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upbdefs.c 3.1% 832 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upbdefs.c 2.4% 648 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upbdefs.c 1.2% 320 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/timestamp.upbdefs.c 1.2% 312 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upbdefs.c 1.1% 304 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/field_mask.upbdefs.c 1.1% 296 [1 Others] 19.1% 24.5Ki JSON 60.4% 14.8Ki upb/json_decode.c 39.6% 9.67Ki upb/json_encode.c 14.7% 18.9Ki Reflection Runtime 81.9% 15.4Ki upb/def.c 18.1% 3.41Ki upb/reflection.c 13.9% 17.8Ki Generated Code 46.9% 8.34Ki bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upb.c 17.6% 3.12Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upb.c 16.4% 2.92Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upb.c 14.7% 2.61Ki bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/descriptor.upb.c 2.0% 356 bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upb.c 1.4% 264 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upb.c 0.5% 88 [2 Others] 0.3% 48 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/any.upb.c 0.3% 48 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upb.c 10.0% 12.9Ki Binary Format 54.0% 6.95Ki upb/decode.c 46.0% 5.92Ki upb/encode.c 8.8% 11.3Ki Core Runtime 66.3% 7.52Ki upb/table.c 17.8% 2.02Ki upb/upb.c 15.9% 1.80Ki upb/msg.c 5.6% 7.19Ki [34 Others] 3.8% 4.94Ki Text Format 100.0% 4.94Ki upb/text_encode.c 3.5% 4.46Ki [section .rodata] 100.0% 128Ki TOTAL So JSON is pulling in 25Ki directly, plus it depends on the 19Ki reflection runtime and the generated reflection. But it is all a la carte: if you don't want JSON, just don't use it and don't link it in. (btw that 7Ki for upb/table.c is to support protobuf maps, I want to make that a la carte also so that you don't pay for it when you don't use maps).

by tijsvd 6y ago

What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

joshuamorton 6y ago

[Also a googler]: PB is used for a wide variety of things, its a lingua-franca of data interchange within Google, but web (frontend) still often uses json. So if I want to have some proto in a database and render it on a UI to a user, somewhere in there I'm probably going to be translating the raw proto to either json or json-like (I'll admit I'm not sure how stuff like grpc-web fits in here, if you can just get proto-serialized bytes passed around in some a blob, but there are some pretty low-overhead json proto encodings that exist anyway).

tijsvd 6y ago
Sure that makes some sense. But it's somewhat ugly as well, from an architecture point of view. Because that argument translates to anything else you might want to do with these objects. In past libraries for C++ I've tried to prevent this kind of coupling by adding generated template methods like "walk(f)" where f would be a templated callable, called with a descriptor and data reference for each field. Any kind of pretty printer or SQL statement can be built that way.

kentonv 6y ago

People implementing public-facing APIs might not be comfortable forcing all the consumers of that API to use Protobuf, but may want to use Protobuf internally within their service. So, Protobuf lets you convert to/from JSON, so that your service can expose a JSON API while maintaining Protobuf schemas and never dealing with a JSON parser directly. The implementation is based on Protobuf's existing reflection features, so if you are already compiling with that enabled, the JSON implementation does not add much code bloat on top of that.

haberman 6y ago

> Don't people use pb so they do not have to deal with JSON? I think that's one reason, but there are others too, like getting generated type-safe accessors and an explicit schema. I've sometimes used protobuf for this reason even when I'm not planning to use binary format. > I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size? Here is a breakdown of a binary that uses upb and links in binary format, JSON format, text format, and some generated protos: VM SIZE --------------- 20.6% 26.5Ki Generated Reflection 45.7% 12.1Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upbdefs.c 39.3% 10.4Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upbdefs.c 5.0% 1.33Ki bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upbdefs.c 3.1% 832 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upbdefs.c 2.4% 648 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upbdefs.c 1.2% 320 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/timestamp.upbdefs.c 1.2% 312 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upbdefs.c 1.1% 304 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/field_mask.upbdefs.c 1.1% 296 [1 Others] 19.1% 24.5Ki JSON 60.4% 14.8Ki upb/json_decode.c 39.6% 9.67Ki upb/json_encode.c 14.7% 18.9Ki Reflection Runtime 81.9% 15.4Ki upb/def.c 18.1% 3.41Ki upb/reflection.c 13.9% 17.8Ki Generated Code 46.9% 8.34Ki bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upb.c 17.6% 3.12Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upb.c 16.4% 2.92Ki bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upb.c 14.7% 2.61Ki bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/descriptor.upb.c 2.0% 356 bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upb.c 1.4% 264 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upb.c 0.5% 88 [2 Others] 0.3% 48 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/any.upb.c 0.3% 48 bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upb.c 10.0% 12.9Ki Binary Format 54.0% 6.95Ki upb/decode.c 46.0% 5.92Ki upb/encode.c 8.8% 11.3Ki Core Runtime 66.3% 7.52Ki upb/table.c 17.8% 2.02Ki upb/upb.c 15.9% 1.80Ki upb/msg.c 5.6% 7.19Ki [34 Others] 3.8% 4.94Ki Text Format 100.0% 4.94Ki upb/text_encode.c 3.5% 4.46Ki [section .rodata] 100.0% 128Ki TOTAL So JSON is pulling in 25Ki directly, plus it depends on the 19Ki reflection runtime and the generated reflection. But it is all a la carte: if you don't want JSON, just don't use it and don't link it in. (btw that 7Ki for upb/table.c is to support protobuf maps, I want to make that a la carte also so that you don't pay for it when you don't use maps).