3 ms·

It's insane that with this proliferation of serialization libraries none of them are still usable for real-world work. Here in 2018, we still have to roll our

It's insane that with this proliferation of serialization libraries none of them are still usable for real-world work. Here in 2018, we still have to roll our own. Libnop has the right idea, but still fails on two important points: a) Versioning is a must-have feature. (And no, some weird non-standard 'table' type is not a replacement.) b) The wire protocol needs to be human-readable ASCII. (Or at least a human-readable ASCII with no performance degradation option must be available.)

coreytabaka 8y ago
The table approach is fairly flexible. Both Protos and FlatBuffers do more or less functionally similar things. What specific deficiency do you see? Take a look at the binary format spec: https://github.com/google/libnop/blob/master/docs/format.md https://github.com/google/libnop/blob/master/docs/format.md The format is designed specifically to provide enough structural information that the binary format can be parsed without knowing the original structure definitions. This makes it easy to write a binary-to-string converter than can grok any payload with minimal complexity. Message observability was a primary goal during development.
- otabdeveloper2 8y ago
  Proto/Flatbuffers are okay for message passing, but many times you don't want to pass messages, you want to serialize C++ structures. There's no way to represent your 1Gb in-memory hashtable as a Flatbuffer, and that is not okay. Writing a binary-to-string converter sucks. You write code once, and debug it every day. Anything that makes debugging harder is humongous pain that shouldn't exist; binary-to-string converters can be written, but let's be real for a moment: this is something that is dead last on the list of business priorities. If your message can be parsed with ad-hoc shell or Python scripts, that's a huge win for everyone.
throwaway080383 8y ago
I can assure you that Protobufs and Thrift are used in the real world.

coreytabaka 8y ago

The table approach is fairly flexible. Both Protos and FlatBuffers do more or less functionally similar things. What specific deficiency do you see? Take a look at the binary format spec: https://github.com/google/libnop/blob/master/docs/format.md https://github.com/google/libnop/blob/master/docs/format.md The format is designed specifically to provide enough structural information that the binary format can be parsed without knowing the original structure definitions. This makes it easy to write a binary-to-string converter than can grok any payload with minimal complexity. Message observability was a primary goal during development.

otabdeveloper2 8y ago
Proto/Flatbuffers are okay for message passing, but many times you don't want to pass messages, you want to serialize C++ structures. There's no way to represent your 1Gb in-memory hashtable as a Flatbuffer, and that is not okay. Writing a binary-to-string converter sucks. You write code once, and debug it every day. Anything that makes debugging harder is humongous pain that shouldn't exist; binary-to-string converters can be written, but let's be real for a moment: this is something that is dead last on the list of business priorities. If your message can be parsed with ad-hoc shell or Python scripts, that's a huge win for everyone.

throwaway080383 8y ago

I can assure you that Protobufs and Thrift are used in the real world.