3 ms·

If I understand the idea of protobufs correctly, the fundamental point is to start with a serialization specification and then generate reader/writer code for t

If I understand the idea of protobufs correctly, the fundamental point is to start with a serialization specification and then generate reader/writer code for that spec in the programming language of your choice. If that is correct, it is very similar to the idea of pattern matching in functional programming compilers. See for example: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.257.6166&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.25... The fundamental idea is that when you have a function definition like f(<pattern>) <- ..... instead of a runtime binary match(<pattern>,<expression>), the compiler generates a function (lets call it fmp) such that fmp(<expression>) = match(<pattern>,<expression>) for any <expression> Ever since I first used pattern matching compilation when I worked on the Hope compiler, I always though that the idea of precompiling static function parameters (or slowly changing ones) was a very under-used idea.

cryptonector 6y ago
Protobufs is just yet another structured data serialization format. There's nothing special to it other than "it's not one of the others". It's a bad reinvention of the wheel though. That's why there's new and better things like flatbuffers. The most important thing to take away from protobufs is that in a field full of literature and prior art, one should not ignore it.
- mumblemumble 6y ago
  For the curious, here's my hot take on the details: Protocol buffers, being older, are far and away the more widely supported format. If you're looking for something like a wire format for RPC, or usability from basically any common language is important to you, protobuf will be the path of least resistance. However, there are some design warts to work around. The most notable one (IMO) is that varints make dealing with numbers a rather delicate subject. Flatbuffers are similar, but made some design decisions allow for a zero-copy implementation. This can be good for performance. Overall, flatbuffers have always struck me as being promising, but language and tooling support isn't nearly as comprehensive, which has always kept me from giving them a serious try at work.
  - mumblemumble 6y ago
    A much later thought, that perhaps speaks to the idea that one size does not fit all: The Apache Arrow project, which is the most ambitious project I know of for sharing data among systems in a performant way, ended up going with a mix of options. Actual data is stored in a project-specific format. When the data are serialized (e.g., to disk or for transport over the wire), headers, schemata, and suchlike are formatted using FlatBuffers. The RPC protocol itself uses Protobuf for the messages.
- kortex 6y ago
  Protobufs aren't perfect, but flatbuffers aren't the holy grail, either. I've found the Protobuf ecosystem way better documented (and yet still lacking), and way more market penetration (due to grpc, no doubt). I've found flatbuf's api ergonomics absolutely dreadful. The python flatbuf library re-parses bytes on each access, which kind of defeats the purpose of zero copy. Maybe it's better in other langs, but python was where I was hoping to reap performance gains and didn't see them.

cryptonector 6y ago

Protobufs is just yet another structured data serialization format. There's nothing special to it other than "it's not one of the others". It's a bad reinvention of the wheel though. That's why there's new and better things like flatbuffers. The most important thing to take away from protobufs is that in a field full of literature and prior art, one should not ignore it.

mumblemumble 6y ago
For the curious, here's my hot take on the details: Protocol buffers, being older, are far and away the more widely supported format. If you're looking for something like a wire format for RPC, or usability from basically any common language is important to you, protobuf will be the path of least resistance. However, there are some design warts to work around. The most notable one (IMO) is that varints make dealing with numbers a rather delicate subject. Flatbuffers are similar, but made some design decisions allow for a zero-copy implementation. This can be good for performance. Overall, flatbuffers have always struck me as being promising, but language and tooling support isn't nearly as comprehensive, which has always kept me from giving them a serious try at work.
- mumblemumble 6y ago
  A much later thought, that perhaps speaks to the idea that one size does not fit all: The Apache Arrow project, which is the most ambitious project I know of for sharing data among systems in a performant way, ended up going with a mix of options. Actual data is stored in a project-specific format. When the data are serialized (e.g., to disk or for transport over the wire), headers, schemata, and suchlike are formatted using FlatBuffers. The RPC protocol itself uses Protobuf for the messages.
kortex 6y ago
Protobufs aren't perfect, but flatbuffers aren't the holy grail, either. I've found the Protobuf ecosystem way better documented (and yet still lacking), and way more market penetration (due to grpc, no doubt). I've found flatbuf's api ergonomics absolutely dreadful. The python flatbuf library re-parses bytes on each access, which kind of defeats the purpose of zero copy. Maybe it's better in other langs, but python was where I was hoping to reap performance gains and didn't see them.