5 ms·

> Protobuffers are shit coz they don't support zero copy and you have to deserialize the whole thing even if you are interested in one field or an outer envelop

> Protobuffers are shit coz they don't support zero copy and you have to deserialize the whole thing even if you are interested in one field or an outer envelope, causing memory churn in your JVMs. Cap'n'proto and flat buffers attack this real problem. The expressivity of the type system is a minor issue, hence no credible competition. I'm not terribly familiar with protobuffers, but I'm a little surprised by this. Some ASN.1 encodings (BER, CER, DER), by contrast, use nested tag-length-value triads. This allows you to skip parts of the message that aren't interesting. (This is, by the way, not that uncommon.) > Required should be enforced at application later not the binary packing layer. It is a property of the version of the code processing the blob, not the blob representation itself. This might depend a bit. Suppose you wanted to make sure that messages could be read (if not exactly decoded) without a schema. The structure of the message could, in principle, include this information in the form of bit-field preambles that indicate the number of fields, extensibility, and so forth. I don't suppose that's strictly necessary for most applications: embedding message structure in message content seems like a bit of an anti-pattern, but I bet you could come up with a use case that makes sense in some context.

kentonv 7y ago
> nested tag-length-value triads. This allows you to skip parts of the message that aren't interesting. Protobuf does that too. But in order to seek horizontally through an array, you need to inspect the tag/length of each element in order to skip to the next element. So you can't rapidly seek through a massive array. In order to allow such seeking, you either need statically-sized elements (which is often too restrictive), or you need pointers to data represented out-of-line. Cap'n Proto uses pointers. Using pointers is pretty uncommon among serialization formats, which is strange considering how ubiquitous they are for in-memory data structures. (Disclosure: I'm the author of Cap'n Proto, and also Protobuf v2.)
- cryptonector 7y ago
  "ASN.1/DER sucks -- everyone knows that. Let's build a new, better thing. How shall we encode things? Oh oh oh! I know! Let's have a type tag, a length, and a value encoding. Perfect! So so much better than ASN.1/DER!!!" Those who don't study the past... Look, ASN.1 is a crappy (but not awful!) schema language, but it supports many many encodings, and some of them are dumb, stupid, and bad, like BER, DER, and CER, and some are clever (PER), and some are awesome (OER), and it even supports things like XML (XER) and even JSON. So what's so bad about ASN.1? Not much, really, just that the first generation of encoding rules (BER/DER/CER) for it were. But no, people don't look. They jump without looking, and then they re-create things, and do it badly. The only thing since ASN.1 that doesn't suck is XDR. That's because XDR very much resembles ASN.1's PER/OER, but with 4-byte units, so XDR is very ergonomic. EDIT: I should also mention flatbuffers as not sucking.
  - kentonv 7y ago
    > Look, ASN.1 is a crappy (but not awful!) schema language, TBH I think it's awful. The type system is woefully overcomplicated, and the syntax is totally unrelated to any popular programming language, making it hard to learn. The fact that there are so many encodings creates confusion for the average developer who frankly cares much more about the schema language accessibility, tooling, API, language support, and documentation than the actual encoding. Protobuf is way, way ahead on all of those compared to ASN.1. > The only thing since ASN.1 that doesn't suck is XDR. Sun XDR, from the 80's? Is that actually newer than ASN.1?
    - cryptonector 7y ago
      > > Look, ASN.1 is a crappy (but not awful!) schema language, > TBH I think it's awful. The type system is woefully overcomplicated, and the syntax is totally unrelated to any popular programming language, making it hard to learn. Is it? How? What is an alternative you like better? > > The only thing since ASN.1 that doesn't suck is XDR. > Sun XDR, from the 80's? Is that actually newer than ASN.1? The first ASN.1 specs are from 1984 (I guess it goes back a bit further). XDR/NFS are from 1986-1987. ASN.1 probably did not inform XDR in the least. Other RPC technologies from that time probably did (thinking of Apollo and such). What's interesting is that even if XDR is completely uninformed by ASN.1, it's essentially a subset of ASN.1 with different syntax and a PER-like encoding with 4-octet unit size and alignment -- the similarities are striking! And even more interesting is that the ASN.1 crowd in 1984 felt that TLV encodings were easy and non-TLV encodings like PER difficult, but XDR shows that non-TLV is not very difficult at all. The lesson of the ASN.1 experience is that TLV == bad, and PER-like == good, though flatbuffers is probably the best. And more than that, the real lesson is that open source tooling is essential. It took too many decades for ASN.1 to have decent open source tooling. Another lesson of the ASN.1 experience is that non-free standards really suck for pervasive and essential technologies. It took way to long for the ITU-T to make the ASN.1 specs available for free downloads. Now, I do understand that the ASN.1 specs are extremely well-written -- it's clear that it cost quite a lot of money to produce them, and somehow that has to be paid for, for the IETF model is much more accessible, and that is much more important than the high quality of specifications that the ITU-T is able to produce (much better than IETF RFCs, IMO).
seriesf 7y ago
You can just skip messages, strings, and byte arrays you don’t need or can’t decode. They are length-prefixed. Also there’s nothing about proto that prevents it from being aliased to its network buffers (zero copy). The GPs complaint stems from their own ignorance, nothing to do with the nature of protos.
- tlarkworthy 7y ago
  https://github.com/protocolbuffers/protobuf/issues/1896 https://github.com/protocolbuffers/protobuf/issues/1896 Ok some of the things I said couldn't be done, can be done, but it's situational

kentonv 7y ago

> nested tag-length-value triads. This allows you to skip parts of the message that aren't interesting. Protobuf does that too. But in order to seek horizontally through an array, you need to inspect the tag/length of each element in order to skip to the next element. So you can't rapidly seek through a massive array. In order to allow such seeking, you either need statically-sized elements (which is often too restrictive), or you need pointers to data represented out-of-line. Cap'n Proto uses pointers. Using pointers is pretty uncommon among serialization formats, which is strange considering how ubiquitous they are for in-memory data structures. (Disclosure: I'm the author of Cap'n Proto, and also Protobuf v2.)

cryptonector 7y ago
"ASN.1/DER sucks -- everyone knows that. Let's build a new, better thing. How shall we encode things? Oh oh oh! I know! Let's have a type tag, a length, and a value encoding. Perfect! So so much better than ASN.1/DER!!!" Those who don't study the past... Look, ASN.1 is a crappy (but not awful!) schema language, but it supports many many encodings, and some of them are dumb, stupid, and bad, like BER, DER, and CER, and some are clever (PER), and some are awesome (OER), and it even supports things like XML (XER) and even JSON. So what's so bad about ASN.1? Not much, really, just that the first generation of encoding rules (BER/DER/CER) for it were. But no, people don't look. They jump without looking, and then they re-create things, and do it badly. The only thing since ASN.1 that doesn't suck is XDR. That's because XDR very much resembles ASN.1's PER/OER, but with 4-byte units, so XDR is very ergonomic. EDIT: I should also mention flatbuffers as not sucking.
- kentonv 7y ago
  > Look, ASN.1 is a crappy (but not awful!) schema language, TBH I think it's awful. The type system is woefully overcomplicated, and the syntax is totally unrelated to any popular programming language, making it hard to learn. The fact that there are so many encodings creates confusion for the average developer who frankly cares much more about the schema language accessibility, tooling, API, language support, and documentation than the actual encoding. Protobuf is way, way ahead on all of those compared to ASN.1. > The only thing since ASN.1 that doesn't suck is XDR. Sun XDR, from the 80's? Is that actually newer than ASN.1?
  - cryptonector 7y ago
    > > Look, ASN.1 is a crappy (but not awful!) schema language, > TBH I think it's awful. The type system is woefully overcomplicated, and the syntax is totally unrelated to any popular programming language, making it hard to learn. Is it? How? What is an alternative you like better? > > The only thing since ASN.1 that doesn't suck is XDR. > Sun XDR, from the 80's? Is that actually newer than ASN.1? The first ASN.1 specs are from 1984 (I guess it goes back a bit further). XDR/NFS are from 1986-1987. ASN.1 probably did not inform XDR in the least. Other RPC technologies from that time probably did (thinking of Apollo and such). What's interesting is that even if XDR is completely uninformed by ASN.1, it's essentially a subset of ASN.1 with different syntax and a PER-like encoding with 4-octet unit size and alignment -- the similarities are striking! And even more interesting is that the ASN.1 crowd in 1984 felt that TLV encodings were easy and non-TLV encodings like PER difficult, but XDR shows that non-TLV is not very difficult at all. The lesson of the ASN.1 experience is that TLV == bad, and PER-like == good, though flatbuffers is probably the best. And more than that, the real lesson is that open source tooling is essential. It took too many decades for ASN.1 to have decent open source tooling. Another lesson of the ASN.1 experience is that non-free standards really suck for pervasive and essential technologies. It took way to long for the ITU-T to make the ASN.1 specs available for free downloads. Now, I do understand that the ASN.1 specs are extremely well-written -- it's clear that it cost quite a lot of money to produce them, and somehow that has to be paid for, for the IETF model is much more accessible, and that is much more important than the high quality of specifications that the ITU-T is able to produce (much better than IETF RFCs, IMO).

seriesf 7y ago

You can just skip messages, strings, and byte arrays you don’t need or can’t decode. They are length-prefixed. Also there’s nothing about proto that prevents it from being aliased to its network buffers (zero copy). The GPs complaint stems from their own ignorance, nothing to do with the nature of protos.

tlarkworthy 7y ago
https://github.com/protocolbuffers/protobuf/issues/1896 https://github.com/protocolbuffers/protobuf/issues/1896 Ok some of the things I said couldn't be done, can be done, but it's situational