4 ms·

I thought the general consensus on protocol buffers was "Meh"?

by sigzero 7y ago

untog 7y ago
You thought wrong?
- sigzero 7y ago
  Seems so.
capableweb 7y ago
I'm wondering if the whole idea behind "general consensus" really work on technical problems. There are so many factors involved with the choices engineers make that you can't say that general consensus says it's "meh" as it really depends on your use case. No tool will ever fit all general problems, so will always be "meh" for someone.
myko 7y ago
I think for large organizations they probably make sense. They're potentially a lot less data per request than JSON. For the average app? Overkill, sure.
- xmprt 7y ago
  I also recently learned about grpcurl that makes using protobufs a lot easier. Still not as easy as json but I think protobufs are the future.
  - SlowRobotAhead 7y ago
    CBOR. Binary JSON with some extra features. I like PB, but it was too hard in my application to sync all of the sides with the same proto as it was developing rapidly. For me schemaless was better, and now we have extremely easy JSON conversion... but did have to write the parser myself. I personally can’t agree on PB being “the future”.
- iainmerrick 7y ago
  Overkill? It doesn’t take a lot of work to add proto buffers to an existing project (if you’re already using a reasonable build system, anyway).
  - tmpz22 7y ago
    If you have weeks to spend optimizing build systems, linters/editors, CI/CD, etc...
manigandham 7y ago
Why would it be? At this point, it's a simple, effective and fast RPC framework with a lot of cross-language support. If you need something like it, there's not many other choices available with this big of an ecosystem. There's even GRPC-Web being worked on to bring it to the browser. That being said, it can be overused just like anything else.
nly 7y ago
It is a bit meh. The IDL and generated code is very ugly, and its neither the fastest nor the most compact (in serialized bytes) format. That said, it's also very widely adopted, and very mature.
abraxas 7y ago
Something like it needs to exist. JSON is stupidly inefficient for large sets of numbers. Except a framework just like grpc has been around for 25 years... but shhhh, don't tell hipsters about IIOP, they are like new parents and need to discover things for themselves.
- fnord77 7y ago
  do you have any links to benchmarks showing json serialization inefficiency for large sets of numbers? I found this which shows json isn't too much worse than others, but it doesn't go into extreme cases https://github.com/eishay/jvm-serializers/wiki https://github.com/eishay/jvm-serializers/wiki
  - abraxas 7y ago
    Anything that represents numeric values as text will be an order of magnitude slower to deserialize than a binary format. I don’t have a published benchmark but in my testing which is easy to replicate I’m storing an array of around 50 million uint64 values in a contiguous block of data to maximize deserialization speed. Just to get a sense of the difference I also tested using a JSON array as a more user friendly storage format and the penalty of deserializing the JSON and populating the array was around 20x in runtime cost in my use case. The platforms I tested on were C# and Java.

I thought the general consensus on protocol buffers was "Meh"?

untog 7y ago

You thought wrong?

sigzero 7y ago
Seems so.

capableweb 7y ago

I'm wondering if the whole idea behind "general consensus" really work on technical problems. There are so many factors involved with the choices engineers make that you can't say that general consensus says it's "meh" as it really depends on your use case. No tool will ever fit all general problems, so will always be "meh" for someone.

myko 7y ago

I think for large organizations they probably make sense. They're potentially a lot less data per request than JSON. For the average app? Overkill, sure.

xmprt 7y ago
I also recently learned about grpcurl that makes using protobufs a lot easier. Still not as easy as json but I think protobufs are the future.
- SlowRobotAhead 7y ago
  CBOR. Binary JSON with some extra features. I like PB, but it was too hard in my application to sync all of the sides with the same proto as it was developing rapidly. For me schemaless was better, and now we have extremely easy JSON conversion... but did have to write the parser myself. I personally can’t agree on PB being “the future”.
iainmerrick 7y ago
Overkill? It doesn’t take a lot of work to add proto buffers to an existing project (if you’re already using a reasonable build system, anyway).
- tmpz22 7y ago
  If you have weeks to spend optimizing build systems, linters/editors, CI/CD, etc...

manigandham 7y ago

Why would it be? At this point, it's a simple, effective and fast RPC framework with a lot of cross-language support. If you need something like it, there's not many other choices available with this big of an ecosystem. There's even GRPC-Web being worked on to bring it to the browser. That being said, it can be overused just like anything else.

nly 7y ago

It is a bit meh. The IDL and generated code is very ugly, and its neither the fastest nor the most compact (in serialized bytes) format. That said, it's also very widely adopted, and very mature.

abraxas 7y ago

Something like it needs to exist. JSON is stupidly inefficient for large sets of numbers. Except a framework just like grpc has been around for 25 years... but shhhh, don't tell hipsters about IIOP, they are like new parents and need to discover things for themselves.

fnord77 7y ago
do you have any links to benchmarks showing json serialization inefficiency for large sets of numbers? I found this which shows json isn't too much worse than others, but it doesn't go into extreme cases https://github.com/eishay/jvm-serializers/wiki https://github.com/eishay/jvm-serializers/wiki
- abraxas 7y ago
  Anything that represents numeric values as text will be an order of magnitude slower to deserialize than a binary format. I don’t have a published benchmark but in my testing which is easy to replicate I’m storing an array of around 50 million uint64 values in a contiguous block of data to maximize deserialization speed. Just to get a sense of the difference I also tested using a JSON array as a more user friendly storage format and the penalty of deserializing the JSON and populating the array was around 20x in runtime cost in my use case. The platforms I tested on were C# and Java.