3 ms·

Cool. When you evaluated all the alternatives, did you stash a pro/con list somewhere on a blog? I’d appreciate reading it

hwbehrens 7y ago
There is a comparison table on the linked GitHub page, with the following formats: Concise (the proposed format), XML, JSON, BSON, CBOR, Messagepack, Cap'n, Protobufs, Flatbuffers, Thrift, and ASN.1. Although not technically a pro/con list, it highlights the evaluation criteria used to compare.
kstenerud 7y ago
I made a short list here [1]. I also kept a design document which explains the choices I made [2]. [1] https://github.com/kstenerud/concise-encoding#comparison-to-other-formats https://github.com/kstenerud/concise-encoding#comparison-to-... [2] https://github.com/kstenerud/concise-encoding/blob/master/design.md https://github.com/kstenerud/concise-encoding/blob/master/de...
- kentonv 7y ago
  Hmm, I think your table is using a different definition of "zero-copy" than is currently common when discussing serialization formats. In your comparison you claim Concise is zero-copy, but it does not appear to be so, at least in the sense that Cap'n Proto and FlatBuffers are. Concise appears to use variable-width integers, tag-value sequences, and 8-byte alignment, all of which make encoded messages unsuitable as in-memory data structures; it looks to me like you have to parse the message into some sort of AST before you could meaningfully operate on the content. "Zero-copy" in the Cap'n Proto / FlatBuffers sense means that there's no need to do any "parsing" because the whole message is efficiently traverseable as-is; e.g. you can mmap() in a very large message and then find and access any one value within it in O(1) time (or perhaps O(log n), if the data structure is nested). This generally requires that primitive values have fixed widths and fields have fixed offsets within their parent object. Digging into your docs it looks like what you really mean by "zero copy" is that individual strings or byte sequences can be used directly without copying them out of the original message buffer. This is a fairly common property of any binary protocol; e.g. Protocol Buffers can do this (albeit without NUL-termination), but in your table you have indicated that it cannot. I would maybe call this "zero-copy strings" to be clearer. On another note, in your table you have indicated that Cap'n Proto doesn't have a "String" type, but this is not true -- Cap'n Proto's "Text" type is a UTF-8 string (and is even NUL-terminated on the wire). (I'm the author of Cap'n Proto.)

hwbehrens 7y ago

There is a comparison table on the linked GitHub page, with the following formats: Concise (the proposed format), XML, JSON, BSON, CBOR, Messagepack, Cap'n, Protobufs, Flatbuffers, Thrift, and ASN.1. Although not technically a pro/con list, it highlights the evaluation criteria used to compare.

kstenerud 7y ago

I made a short list here [1]. I also kept a design document which explains the choices I made [2]. [1] https://github.com/kstenerud/concise-encoding#comparison-to-other-formats https://github.com/kstenerud/concise-encoding#comparison-to-... [2] https://github.com/kstenerud/concise-encoding/blob/master/design.md https://github.com/kstenerud/concise-encoding/blob/master/de...

kentonv 7y ago
Hmm, I think your table is using a different definition of "zero-copy" than is currently common when discussing serialization formats. In your comparison you claim Concise is zero-copy, but it does not appear to be so, at least in the sense that Cap'n Proto and FlatBuffers are. Concise appears to use variable-width integers, tag-value sequences, and 8-byte alignment, all of which make encoded messages unsuitable as in-memory data structures; it looks to me like you have to parse the message into some sort of AST before you could meaningfully operate on the content. "Zero-copy" in the Cap'n Proto / FlatBuffers sense means that there's no need to do any "parsing" because the whole message is efficiently traverseable as-is; e.g. you can mmap() in a very large message and then find and access any one value within it in O(1) time (or perhaps O(log n), if the data structure is nested). This generally requires that primitive values have fixed widths and fields have fixed offsets within their parent object. Digging into your docs it looks like what you really mean by "zero copy" is that individual strings or byte sequences can be used directly without copying them out of the original message buffer. This is a fairly common property of any binary protocol; e.g. Protocol Buffers can do this (albeit without NUL-termination), but in your table you have indicated that it cannot. I would maybe call this "zero-copy strings" to be clearer. On another note, in your table you have indicated that Cap'n Proto doesn't have a "String" type, but this is not true -- Cap'n Proto's "Text" type is a UTF-8 string (and is even NUL-terminated on the wire). (I'm the author of Cap'n Proto.)