3 ms·

> compared to JSON with its anemic type support Am I the only one who wants to keep serialization as far from my type system as possible? I want to have my in

> compared to JSON with its anemic type support Am I the only one who wants to keep serialization as far from my type system as possible? I want to have my internal data model for my software, and when I want to serialize it, I should be able to do it in any number of different ways (redacting values, using HATEOAS links for REST routes, using a lossless format for storage) depending on the situation. Protobuf couples data modeling and data serialization into a single inseparable concern, and it becomes difficult to do anything else once you start using it.

com2kid 8y ago
It is a performance trade off. Sure you can use JSON and make everything strings and do something like { type: 'int8', value: '52' } but then parsing out the data ends up being a huge overhead. I once saw an XML variant of this, where someone decided to make the ultimate Distributed Computing System and serialize every single function call, take up over 1/2 of a CPU core for de/serialization for just the one app running on that machine. While I could admire the purity of the design, it was utterly insane as an actual thing to bring into the world. Protobuf is a good mixture of "strict typing so everyone is on the same page", "freedom to do stuff", and "perf isn't horrible." There are other encoding systems out there that offer even more freedom, e.g. cap'n'proto, and better performance, but with other trade offs. On the opposite side of things, my team had been serializing straight C structs out over the wire, but every field addition was a breaking change, and communicating the changes to our structures across teams was a nightmare of meetings and "has your team merged the changes Bob made so we can roll out our new format yet?" We needed 3 teams to roll out changes at the same time! With protobufs, we were able to make changes to our wire format incredibly rapidly, we had a nice source control managed asset that defined our format, there weren't any confusions as to how data was laid out, and clients using older versions of our definitions just missed out on newer features, nothing actually broke. It was an insanely large improvement, and honestly for the managed platforms, the performance wasn't appreciably worse than trying to convince Java to read in uint8s and write out uint8s. My team was working on an embedded platform with RAM measured in kilobytes, and it was worth us eating the overhead just to get rid of the countless meetings we had to hold whenever we made a change to any of our structures.
lima 8y ago
Not true - protobuf has a JSON representation: https://developers.google.com/protocol-buffers/docs/proto3#json https://developers.google.com/protocol-buffers/docs/proto3#j...
joshuamorton 8y ago
>Protobuf couples data modeling and data serialization into a single inseparable concern, and it becomes difficult to do anything else once you start using it. This is only true if you decide that your storage proto and wire proto are the same. That's not at all necessary (and generally not recommended). More to the point, FieldMask exists in proto, so redaction is fully supported, though I rarely see them used.
randomdata 8y ago
Your internal data model should be separate from protobufs. Protobufs are for modelling the contract between parties of the serialized content so that the producer knows what to produce and consumer knows what data to expect. Only when you have a need to serialize your internal data model into protobufs would you touch protobufs, just like you would with any other serialization format. The structures that are generated by the protobuf tool are simply there to help with transforming your internal state into the format needed to satisfy the shared contract, allowing your language's compiler to tell you if you have violated that contract. Theoretically you could produce/consume protobufs without them, but they are provided as a convenience so that you can deal with serialization transforms directly in your language of choice instead of banging bytes.
- ionforce 8y ago
  I think most people don't take the time to discriminate between their models and protobuf's. They see automagically created data models and think WELP LET'S USE THIS EVERYWHERE.

com2kid 8y ago

It is a performance trade off. Sure you can use JSON and make everything strings and do something like { type: 'int8', value: '52' } but then parsing out the data ends up being a huge overhead. I once saw an XML variant of this, where someone decided to make the ultimate Distributed Computing System and serialize every single function call, take up over 1/2 of a CPU core for de/serialization for just the one app running on that machine. While I could admire the purity of the design, it was utterly insane as an actual thing to bring into the world. Protobuf is a good mixture of "strict typing so everyone is on the same page", "freedom to do stuff", and "perf isn't horrible." There are other encoding systems out there that offer even more freedom, e.g. cap'n'proto, and better performance, but with other trade offs. On the opposite side of things, my team had been serializing straight C structs out over the wire, but every field addition was a breaking change, and communicating the changes to our structures across teams was a nightmare of meetings and "has your team merged the changes Bob made so we can roll out our new format yet?" We needed 3 teams to roll out changes at the same time! With protobufs, we were able to make changes to our wire format incredibly rapidly, we had a nice source control managed asset that defined our format, there weren't any confusions as to how data was laid out, and clients using older versions of our definitions just missed out on newer features, nothing actually broke. It was an insanely large improvement, and honestly for the managed platforms, the performance wasn't appreciably worse than trying to convince Java to read in uint8s and write out uint8s. My team was working on an embedded platform with RAM measured in kilobytes, and it was worth us eating the overhead just to get rid of the countless meetings we had to hold whenever we made a change to any of our structures.

lima 8y ago

Not true - protobuf has a JSON representation: https://developers.google.com/protocol-buffers/docs/proto3#json https://developers.google.com/protocol-buffers/docs/proto3#j...

joshuamorton 8y ago

>Protobuf couples data modeling and data serialization into a single inseparable concern, and it becomes difficult to do anything else once you start using it. This is only true if you decide that your storage proto and wire proto are the same. That's not at all necessary (and generally not recommended). More to the point, FieldMask exists in proto, so redaction is fully supported, though I rarely see them used.

randomdata 8y ago

Your internal data model should be separate from protobufs. Protobufs are for modelling the contract between parties of the serialized content so that the producer knows what to produce and consumer knows what data to expect. Only when you have a need to serialize your internal data model into protobufs would you touch protobufs, just like you would with any other serialization format. The structures that are generated by the protobuf tool are simply there to help with transforming your internal state into the format needed to satisfy the shared contract, allowing your language's compiler to tell you if you have violated that contract. Theoretically you could produce/consume protobufs without them, but they are provided as a convenience so that you can deal with serialization transforms directly in your language of choice instead of banging bytes.

ionforce 8y ago
I think most people don't take the time to discriminate between their models and protobuf's. They see automagically created data models and think WELP LET'S USE THIS EVERYWHERE.