5 ms·

I generally find Protobuf the serialization format to have no benefits over JSON. Although Protobuf is often thought of as being faster to process and more comp

by meisel 4y ago

I generally find Protobuf the serialization format to have no benefits over JSON. Although Protobuf is often thought of as being faster to process and more compact, the reality is that it's usually not (at least as long as you're not working in a systems language like Rust or C++). It's not much faster than JSON to process because in the context of higher-level languages, the time to create usable objects out of the parsed input (which is independent of the serialized data they came from) is usually far larger than the time to parse the payload. And as for being more compact, if you gzip a Protobuf payload and its JSON equivalent, IME they end up being pretty close in size (sometimes the JSON version is actually smaller).

mkoryak 4y ago
What about the benefit of being able to represent data types like Date, double vs float vs byte etc ?
- klysm 4y ago
  ISO8601 gets you pretty far with dates and you can always quote your numbers - most serialization frameworks support that
zeroxfe 4y ago
We migrated many of our critical protocols from JSON to protobuf, and the experience has been very positive. As your system gets bigger, and your teams get bigger, the value of a typed schema has a multiplicative effect on both reliability and productivity. In addition, a lot of basic features around managing transport, like load balancing, health checking, deadlines, etc. come out of the box. IMO, performance is the least of the reasons to use protobufs. And if you're working on simple systems in small teams (or solo), then JSON is a better fit (because it's simpler to understand, debug, and iterate with.) gRPC is definitely not a one-size-fits-all.
- nwah1 4y ago
  Swagger/OpenAPI offers a typed API in JSON (along with generated clients just like gRPC). The complaints about protobuf as a serialization format remain valid, and you don't need it to have types. Also the reability of JSON in-transit is another factor to consider.
- dataflow 4y ago
  I think the parent comment was talking about the benefits of the protobuf serialization format, not the benefits of the protobuf schema.
- fizwidget 4y ago
  You can get the benefits of a typed schema with GraphQL, while still using JSON under the hood.
- deepstack 4y ago
  >As your system gets bigger, and your teams get bigger, the value of a typed schema has a multiplicative effect on both reliability and productivity. I often see this in arguments using in TypeScript vs plain JS debate. Is this really true in practice? Be good to get some more examples on this.
  - JohnHaugeland 4y ago
    It's extremely well documented
- bobbylarrybobby 4y ago
  Do you really need a typed schema for transmission if you have types on the serialization and deserialization ends? I'm thinking of something like Rust's serde that derives (de)serialization logic just from the type, so it can't serialize bad data and won't deserialize bad data.
deleted 4y ago
[deleted]
Rebelgecko 4y ago
IMO the advantages of protos are mostly related to having firmer typing on your data e.g. instead of writing myJsonPayload["magic string foo"][["magic string bae"], you call methods that actually exist like myMsg.getFoo().getBar(). This is especially helpful if someone who isn't you personally is also writing code that interacts with the data. I suppose that can slow down iteration, but for a decent sized project IMO it's a worthy tradeoff. Another benefit is having nonhacky representations of numbers that don't exist in standard-compliant JSON, like 0 and infinity. It's also handy to be able to embed comments in your data, which wasn't doable last time I used JSON.
- chii 4y ago
  how do you embed comments in protocolbuf data? It's binary! And if you are adding a field specifically for comments, you could do the same in json. For example, a field that has a question mark as the prefix is a comment field (a convention i like). The advantage of protocol buf is, i think, the version being built in. You can make it backwards compatible to older clients. While the same could be done in json, it's manual and requires work and careful design.
  - Rebelgecko 4y ago
    >how do you embed comments in protocolbuf data? It's binary! In addition to the binary format, every proto message can also be represented in a human readable form[1]. This is not as efficient and probably not a good idea for data being sent over the wire as-is.. especially if it is only being written and read by computers. However if you have small amounts of data at rest (like input for a unit test) it can be handy to add some comments explaining anything non-obvious. On top of that, the proto's schema itself can have comments, and those comments can trickle down into the code generation for a given language so that you can read them when working with your messages. Maybe that's also possible with JSON schemas but idk what their tooling ecosystem is like [1]: https://developers.google.com/protocol-buffers/docs/text-format-spec https://developers.google.com/protocol-buffers/docs/text-for...
  - 8n4vidtmkvmk 4y ago
    prototext supports comments. useful for test data and configurations
briHass 4y ago
I've found the opposite, but maybe I'm typically using what you define as a 'systems language' - usually .NET/C#. I really like Protobuf as a pure serializer (not interchange format) to bytes for caching (Redis/Memcached). I had to prove it, again, recently to myself after all the performance improvements made to Json serialization in .NET. Using some collections of real-world objects, with a serialized size ranging between 50-500K, PB was 2x faster than Json, and had payloads about 1/2 the size. Applying GZIP in the serialization brought the JSON size down significantly, but still not as small as PB. PB only compressed slightly, as expected. However, the time for compression made JSON(gzip) almost 3X slower. Now, depending on your network speeds and payload size, maybe the time for compression makes sense for remote/distributed caches, but in that case, why wouldn't I use the smaller PB to begin with.
jillesvangurp 4y ago
It makes sense for some use cases but the vast majority of use cases, parsing overhead is simply not a concern. Mobile phones are fast, networks have plenty of bandwidth (and the savings are marginal), parsers are pretty good. But done right, binary protocols are sometimes worth the marginal savings they provide. We switched over one of our APIs to use CBOR instead of json. It's a search API that we hit a lot and I wanted to cut down on the bytesize of the responses a little. The savings are not that impressive. But I'll take 10% when i can get it. Otherwise, this was a pretty simple change. We use kotlinx serialization in a multi-platform library. Basically, all we did is configure it to use CBOR instead of json. https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/formats.md https://github.com/Kotlin/kotlinx.serialization/blob/master/... Half hour job. Haven't looked at it since; just works. It supports protobuf as well but it looked like more hassle to set up so we went with CBOR instead.

by meisel 4y ago

mkoryak 4y ago

What about the benefit of being able to represent data types like Date, double vs float vs byte etc ?

klysm 4y ago
ISO8601 gets you pretty far with dates and you can always quote your numbers - most serialization frameworks support that

zeroxfe 4y ago

We migrated many of our critical protocols from JSON to protobuf, and the experience has been very positive. As your system gets bigger, and your teams get bigger, the value of a typed schema has a multiplicative effect on both reliability and productivity. In addition, a lot of basic features around managing transport, like load balancing, health checking, deadlines, etc. come out of the box. IMO, performance is the least of the reasons to use protobufs. And if you're working on simple systems in small teams (or solo), then JSON is a better fit (because it's simpler to understand, debug, and iterate with.) gRPC is definitely not a one-size-fits-all.

nwah1 4y ago
Swagger/OpenAPI offers a typed API in JSON (along with generated clients just like gRPC). The complaints about protobuf as a serialization format remain valid, and you don't need it to have types. Also the reability of JSON in-transit is another factor to consider.
dataflow 4y ago
I think the parent comment was talking about the benefits of the protobuf serialization format, not the benefits of the protobuf schema.
fizwidget 4y ago
You can get the benefits of a typed schema with GraphQL, while still using JSON under the hood.
deepstack 4y ago
>As your system gets bigger, and your teams get bigger, the value of a typed schema has a multiplicative effect on both reliability and productivity. I often see this in arguments using in TypeScript vs plain JS debate. Is this really true in practice? Be good to get some more examples on this.
- JohnHaugeland 4y ago
  It's extremely well documented
bobbylarrybobby 4y ago
Do you really need a typed schema for transmission if you have types on the serialization and deserialization ends? I'm thinking of something like Rust's serde that derives (de)serialization logic just from the type, so it can't serialize bad data and won't deserialize bad data.

deleted 4y ago

[deleted]

Rebelgecko 4y ago

IMO the advantages of protos are mostly related to having firmer typing on your data e.g. instead of writing myJsonPayload["magic string foo"][["magic string bae"], you call methods that actually exist like myMsg.getFoo().getBar(). This is especially helpful if someone who isn't you personally is also writing code that interacts with the data. I suppose that can slow down iteration, but for a decent sized project IMO it's a worthy tradeoff. Another benefit is having nonhacky representations of numbers that don't exist in standard-compliant JSON, like 0 and infinity. It's also handy to be able to embed comments in your data, which wasn't doable last time I used JSON.

chii 4y ago
how do you embed comments in protocolbuf data? It's binary! And if you are adding a field specifically for comments, you could do the same in json. For example, a field that has a question mark as the prefix is a comment field (a convention i like). The advantage of protocol buf is, i think, the version being built in. You can make it backwards compatible to older clients. While the same could be done in json, it's manual and requires work and careful design.
- Rebelgecko 4y ago
  >how do you embed comments in protocolbuf data? It's binary! In addition to the binary format, every proto message can also be represented in a human readable form[1]. This is not as efficient and probably not a good idea for data being sent over the wire as-is.. especially if it is only being written and read by computers. However if you have small amounts of data at rest (like input for a unit test) it can be handy to add some comments explaining anything non-obvious. On top of that, the proto's schema itself can have comments, and those comments can trickle down into the code generation for a given language so that you can read them when working with your messages. Maybe that's also possible with JSON schemas but idk what their tooling ecosystem is like [1]: https://developers.google.com/protocol-buffers/docs/text-format-spec https://developers.google.com/protocol-buffers/docs/text-for...
- 8n4vidtmkvmk 4y ago
  prototext supports comments. useful for test data and configurations

briHass 4y ago

I've found the opposite, but maybe I'm typically using what you define as a 'systems language' - usually .NET/C#. I really like Protobuf as a pure serializer (not interchange format) to bytes for caching (Redis/Memcached). I had to prove it, again, recently to myself after all the performance improvements made to Json serialization in .NET. Using some collections of real-world objects, with a serialized size ranging between 50-500K, PB was 2x faster than Json, and had payloads about 1/2 the size. Applying GZIP in the serialization brought the JSON size down significantly, but still not as small as PB. PB only compressed slightly, as expected. However, the time for compression made JSON(gzip) almost 3X slower. Now, depending on your network speeds and payload size, maybe the time for compression makes sense for remote/distributed caches, but in that case, why wouldn't I use the smaller PB to begin with.

jillesvangurp 4y ago

It makes sense for some use cases but the vast majority of use cases, parsing overhead is simply not a concern. Mobile phones are fast, networks have plenty of bandwidth (and the savings are marginal), parsers are pretty good. But done right, binary protocols are sometimes worth the marginal savings they provide. We switched over one of our APIs to use CBOR instead of json. It's a search API that we hit a lot and I wanted to cut down on the bytesize of the responses a little. The savings are not that impressive. But I'll take 10% when i can get it. Otherwise, this was a pretty simple change. We use kotlinx serialization in a multi-platform library. Basically, all we did is configure it to use CBOR instead of json. https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/formats.md https://github.com/Kotlin/kotlinx.serialization/blob/master/... Half hour job. Haven't looked at it since; just works. It supports protobuf as well but it looked like more hassle to set up so we went with CBOR instead.