3 ms·

> nested tag-length-value triads. This allows you to skip parts of the message that aren't interesting. Protobuf does that too. But in order to seek horizontal

> nested tag-length-value triads. This allows you to skip parts of the message that aren't interesting. Protobuf does that too. But in order to seek horizontally through an array, you need to inspect the tag/length of each element in order to skip to the next element. So you can't rapidly seek through a massive array. In order to allow such seeking, you either need statically-sized elements (which is often too restrictive), or you need pointers to data represented out-of-line. Cap'n Proto uses pointers. Using pointers is pretty uncommon among serialization formats, which is strange considering how ubiquitous they are for in-memory data structures. (Disclosure: I'm the author of Cap'n Proto, and also Protobuf v2.)

cryptonector 7y ago
"ASN.1/DER sucks -- everyone knows that. Let's build a new, better thing. How shall we encode things? Oh oh oh! I know! Let's have a type tag, a length, and a value encoding. Perfect! So so much better than ASN.1/DER!!!" Those who don't study the past... Look, ASN.1 is a crappy (but not awful!) schema language, but it supports many many encodings, and some of them are dumb, stupid, and bad, like BER, DER, and CER, and some are clever (PER), and some are awesome (OER), and it even supports things like XML (XER) and even JSON. So what's so bad about ASN.1? Not much, really, just that the first generation of encoding rules (BER/DER/CER) for it were. But no, people don't look. They jump without looking, and then they re-create things, and do it badly. The only thing since ASN.1 that doesn't suck is XDR. That's because XDR very much resembles ASN.1's PER/OER, but with 4-byte units, so XDR is very ergonomic. EDIT: I should also mention flatbuffers as not sucking.
- kentonv 7y ago
  > Look, ASN.1 is a crappy (but not awful!) schema language, TBH I think it's awful. The type system is woefully overcomplicated, and the syntax is totally unrelated to any popular programming language, making it hard to learn. The fact that there are so many encodings creates confusion for the average developer who frankly cares much more about the schema language accessibility, tooling, API, language support, and documentation than the actual encoding. Protobuf is way, way ahead on all of those compared to ASN.1. > The only thing since ASN.1 that doesn't suck is XDR. Sun XDR, from the 80's? Is that actually newer than ASN.1?
  - cryptonector 7y ago
    > > Look, ASN.1 is a crappy (but not awful!) schema language, > TBH I think it's awful. The type system is woefully overcomplicated, and the syntax is totally unrelated to any popular programming language, making it hard to learn. Is it? How? What is an alternative you like better? > > The only thing since ASN.1 that doesn't suck is XDR. > Sun XDR, from the 80's? Is that actually newer than ASN.1? The first ASN.1 specs are from 1984 (I guess it goes back a bit further). XDR/NFS are from 1986-1987. ASN.1 probably did not inform XDR in the least. Other RPC technologies from that time probably did (thinking of Apollo and such). What's interesting is that even if XDR is completely uninformed by ASN.1, it's essentially a subset of ASN.1 with different syntax and a PER-like encoding with 4-octet unit size and alignment -- the similarities are striking! And even more interesting is that the ASN.1 crowd in 1984 felt that TLV encodings were easy and non-TLV encodings like PER difficult, but XDR shows that non-TLV is not very difficult at all. The lesson of the ASN.1 experience is that TLV == bad, and PER-like == good, though flatbuffers is probably the best. And more than that, the real lesson is that open source tooling is essential. It took too many decades for ASN.1 to have decent open source tooling. Another lesson of the ASN.1 experience is that non-free standards really suck for pervasive and essential technologies. It took way to long for the ITU-T to make the ASN.1 specs available for free downloads. Now, I do understand that the ASN.1 specs are extremely well-written -- it's clear that it cost quite a lot of money to produce them, and somehow that has to be paid for, for the IETF model is much more accessible, and that is much more important than the high quality of specifications that the ITU-T is able to produce (much better than IETF RFCs, IMO).
    - kentonv 7y ago
      > Is it? How? There's like a million built-in types and too many unnecessary options for specifying constraints. > What is an alternative you like better? I think Protobuf has a simpler, more practical type system and more accessible syntax compared to ASN.1. Cap'n Proto is very close to Protobuf in terms of type system and syntax, but adds some polish on both. (But Cap'n Proto is my own design, so obviously I think it's the best.) In terms of encoding, Cap'n Proto is, of course, completely different from Protobuf. I guess it is closer to PER and OER... but not particularly close.

cryptonector 7y ago

"ASN.1/DER sucks -- everyone knows that. Let's build a new, better thing. How shall we encode things? Oh oh oh! I know! Let's have a type tag, a length, and a value encoding. Perfect! So so much better than ASN.1/DER!!!" Those who don't study the past... Look, ASN.1 is a crappy (but not awful!) schema language, but it supports many many encodings, and some of them are dumb, stupid, and bad, like BER, DER, and CER, and some are clever (PER), and some are awesome (OER), and it even supports things like XML (XER) and even JSON. So what's so bad about ASN.1? Not much, really, just that the first generation of encoding rules (BER/DER/CER) for it were. But no, people don't look. They jump without looking, and then they re-create things, and do it badly. The only thing since ASN.1 that doesn't suck is XDR. That's because XDR very much resembles ASN.1's PER/OER, but with 4-byte units, so XDR is very ergonomic. EDIT: I should also mention flatbuffers as not sucking.

kentonv 7y ago
> Look, ASN.1 is a crappy (but not awful!) schema language, TBH I think it's awful. The type system is woefully overcomplicated, and the syntax is totally unrelated to any popular programming language, making it hard to learn. The fact that there are so many encodings creates confusion for the average developer who frankly cares much more about the schema language accessibility, tooling, API, language support, and documentation than the actual encoding. Protobuf is way, way ahead on all of those compared to ASN.1. > The only thing since ASN.1 that doesn't suck is XDR. Sun XDR, from the 80's? Is that actually newer than ASN.1?
- cryptonector 7y ago
  > > Look, ASN.1 is a crappy (but not awful!) schema language, > TBH I think it's awful. The type system is woefully overcomplicated, and the syntax is totally unrelated to any popular programming language, making it hard to learn. Is it? How? What is an alternative you like better? > > The only thing since ASN.1 that doesn't suck is XDR. > Sun XDR, from the 80's? Is that actually newer than ASN.1? The first ASN.1 specs are from 1984 (I guess it goes back a bit further). XDR/NFS are from 1986-1987. ASN.1 probably did not inform XDR in the least. Other RPC technologies from that time probably did (thinking of Apollo and such). What's interesting is that even if XDR is completely uninformed by ASN.1, it's essentially a subset of ASN.1 with different syntax and a PER-like encoding with 4-octet unit size and alignment -- the similarities are striking! And even more interesting is that the ASN.1 crowd in 1984 felt that TLV encodings were easy and non-TLV encodings like PER difficult, but XDR shows that non-TLV is not very difficult at all. The lesson of the ASN.1 experience is that TLV == bad, and PER-like == good, though flatbuffers is probably the best. And more than that, the real lesson is that open source tooling is essential. It took too many decades for ASN.1 to have decent open source tooling. Another lesson of the ASN.1 experience is that non-free standards really suck for pervasive and essential technologies. It took way to long for the ITU-T to make the ASN.1 specs available for free downloads. Now, I do understand that the ASN.1 specs are extremely well-written -- it's clear that it cost quite a lot of money to produce them, and somehow that has to be paid for, for the IETF model is much more accessible, and that is much more important than the high quality of specifications that the ITU-T is able to produce (much better than IETF RFCs, IMO).
  - kentonv 7y ago
    > Is it? How? There's like a million built-in types and too many unnecessary options for specifying constraints. > What is an alternative you like better? I think Protobuf has a simpler, more practical type system and more accessible syntax compared to ASN.1. Cap'n Proto is very close to Protobuf in terms of type system and syntax, but adds some polish on both. (But Cap'n Proto is my own design, so obviously I think it's the best.) In terms of encoding, Cap'n Proto is, of course, completely different from Protobuf. I guess it is closer to PER and OER... but not particularly close.