4 ms·

Isn't this the point of things like protobuf and thrift? It seems like everyone beyond a certain scale sends data around in some binary format rather than JSON.

bluejekyll 3y ago
Yes, I’m wondering if people have bad experiences with CBOR, thrift, flatbuffers, protobuf, messagepack, etc. There are a huge number. I think Protobuf might be the best choice across all languages at this point, maybe? But that’s my question, if you choose a binary format, what are the issues to look out for? Example, I work with Java, Rust, Go, and Python at work. I have specifically had issues with Avro (not my choice) which is really well supported in Java, but not much else.
- f_devd 3y ago
  I dislike protobuf for the reason that the generated code is often quite bad (in terms of the interface in the language you are using) and in external projects it's often a manual step (sometimes with additional headache of protoc2 vs protoc3) to just build the project. In terms of message specification it's quite good though.
  - packetlost 3y ago
    This is the reason we opted for building our own code gen system instead of Thrift or Protobuf: they both generate really ugly code in the languages we wanted to use
- mdtusz 3y ago
  Still early days, but we've been using CBOR instead of JSON lately at work for interfaces that have "settled" and it's been great. Means that you can shake out the early integration issues using human readable JSON, then just switch the ser/de once it's all playing nice. Binary data support is pretty nice too for avoiding multipart request bodies.
- importantbrian 3y ago
  Ah, I see what you were saying now. Yeah, I don't have direct experience with those binary formats so I can't say. I'm not an Avro fan either but there are other binary formats like parquet that are easier to work with and at least in my corner of the tech world seems to have become the de facto standard.
alias_neo 3y ago
I would have suggested ProtoBuf too. That said, there was a comment in their post about not wanting to tightly couple the Go and Rust since they weren't sure they could accurately represent their structures in both langs. I don't know anything about this turbo tool, but it seems relatively low-throughput, and they intend to migrate it all to Rust anyway, so it's probably not a big deal. That said; I'd have done this with binary comms and not JSON, ProtoBuf or not.
iamcalledrob 3y ago
I'm using protobuf for exactly this use-case, but I'd say that it's clearly designed as a network format first, and makes trade-offs as such. For example, it prioritizes backwards and forwards compatibility -- which is not a concern for IPC where you control both ends. So no real optional or required fields. Comparing structures (Go) is awkward and uses reflection. Structs embed a mutex and preserve bytes for unknown fields etc... MessagePack seems to make a better set of tradeoffs by comparison.

bluejekyll 3y ago

Yes, I’m wondering if people have bad experiences with CBOR, thrift, flatbuffers, protobuf, messagepack, etc. There are a huge number. I think Protobuf might be the best choice across all languages at this point, maybe? But that’s my question, if you choose a binary format, what are the issues to look out for? Example, I work with Java, Rust, Go, and Python at work. I have specifically had issues with Avro (not my choice) which is really well supported in Java, but not much else.

f_devd 3y ago
I dislike protobuf for the reason that the generated code is often quite bad (in terms of the interface in the language you are using) and in external projects it's often a manual step (sometimes with additional headache of protoc2 vs protoc3) to just build the project. In terms of message specification it's quite good though.
- packetlost 3y ago
  This is the reason we opted for building our own code gen system instead of Thrift or Protobuf: they both generate really ugly code in the languages we wanted to use
mdtusz 3y ago
Still early days, but we've been using CBOR instead of JSON lately at work for interfaces that have "settled" and it's been great. Means that you can shake out the early integration issues using human readable JSON, then just switch the ser/de once it's all playing nice. Binary data support is pretty nice too for avoiding multipart request bodies.
importantbrian 3y ago
Ah, I see what you were saying now. Yeah, I don't have direct experience with those binary formats so I can't say. I'm not an Avro fan either but there are other binary formats like parquet that are easier to work with and at least in my corner of the tech world seems to have become the de facto standard.

alias_neo 3y ago

I would have suggested ProtoBuf too. That said, there was a comment in their post about not wanting to tightly couple the Go and Rust since they weren't sure they could accurately represent their structures in both langs. I don't know anything about this turbo tool, but it seems relatively low-throughput, and they intend to migrate it all to Rust anyway, so it's probably not a big deal. That said; I'd have done this with binary comms and not JSON, ProtoBuf or not.

iamcalledrob 3y ago

I'm using protobuf for exactly this use-case, but I'd say that it's clearly designed as a network format first, and makes trade-offs as such. For example, it prioritizes backwards and forwards compatibility -- which is not a concern for IPC where you control both ends. So no real optional or required fields. Comparing structures (Go) is awkward and uses reflection. Structs embed a mutex and preserve bytes for unknown fields etc... MessagePack seems to make a better set of tradeoffs by comparison.