4 ms·

This is another reason why I'm developing an ad-hoc data representation with strong typing. JSON was almost right, but failed because there are too many fundame

This is another reason why I'm developing an ad-hoc data representation with strong typing. JSON was almost right, but failed because there are too many fundamental data types that cannot be represented in a portable way. A general purpose ad-hoc data format with all the fundamental types, that doesn't require endless text parsing but can be converted to a 1:1 text format for human viewing, would be helpful in so many situations, including serialization for piping between unix tools. https://github.com/kstenerud/concise-encoding#example https://github.com/kstenerud/concise-encoding#example

pinkfoot 7y ago
ASN.1 ?
- kstenerud 7y ago
  Too complicated, too many extensions, too bloated on the wire.
  - C1sc0cat 7y ago
    What you didn't have fun decoding OSI protocols :-) I was a bit of a wuss and had to use x.409 dumps, my boss could look at the raw data packets and debug problems.
  - pinkfoot 7y ago
    PER is bloated?
throwaway3157 7y ago
Cool. When you evaluated all the alternatives, did you stash a pro/con list somewhere on a blog? I’d appreciate reading it
- hwbehrens 7y ago
  There is a comparison table on the linked GitHub page, with the following formats: Concise (the proposed format), XML, JSON, BSON, CBOR, Messagepack, Cap'n, Protobufs, Flatbuffers, Thrift, and ASN.1. Although not technically a pro/con list, it highlights the evaluation criteria used to compare.
- kstenerud 7y ago
  I made a short list here [1]. I also kept a design document which explains the choices I made [2]. [1] https://github.com/kstenerud/concise-encoding#comparison-to-other-formats https://github.com/kstenerud/concise-encoding#comparison-to-... [2] https://github.com/kstenerud/concise-encoding/blob/master/design.md https://github.com/kstenerud/concise-encoding/blob/master/de...
  - kentonv 7y ago
    Hmm, I think your table is using a different definition of "zero-copy" than is currently common when discussing serialization formats. In your comparison you claim Concise is zero-copy, but it does not appear to be so, at least in the sense that Cap'n Proto and FlatBuffers are. Concise appears to use variable-width integers, tag-value sequences, and 8-byte alignment, all of which make encoded messages unsuitable as in-memory data structures; it looks to me like you have to parse the message into some sort of AST before you could meaningfully operate on the content. "Zero-copy" in the Cap'n Proto / FlatBuffers sense means that there's no need to do any "parsing" because the whole message is efficiently traverseable as-is; e.g. you can mmap() in a very large message and then find and access any one value within it in O(1) time (or perhaps O(log n), if the data structure is nested). This generally requires that primitive values have fixed widths and fields have fixed offsets within their parent object. Digging into your docs it looks like what you really mean by "zero copy" is that individual strings or byte sequences can be used directly without copying them out of the original message buffer. This is a fairly common property of any binary protocol; e.g. Protocol Buffers can do this (albeit without NUL-termination), but in your table you have indicated that it cannot. I would maybe call this "zero-copy strings" to be clearer. On another note, in your table you have indicated that Cap'n Proto doesn't have a "String" type, but this is not true -- Cap'n Proto's "Text" type is a UTF-8 string (and is even NUL-terminated on the wire). (I'm the author of Cap'n Proto.)
carry_bit 7y ago
If JSON was almost right, why not EDN? https://github.com/edn-format/edn https://github.com/edn-format/edn
- kstenerud 7y ago
  EDN is better than JSON, but it's still text only, and not fully specified.
tasogare 7y ago
The comparison table doesn't seem fair, especially on the XML column. In particular, it is stated that XML doesn't support decimal and float, whereas XSD in fact defines 19 primitive data types including decimal and float. List are also supported implicitly because child nodes of an element are ordered. Moreover, the claim that XML doesn't support metadata is devious because any file can precisely be extended with any additional metadata (possibly in their own namespace), without client breaking if done well. XML strength also lies in its extensibility, and its large number of compliant implementations, which goes way way beyond a single uncompleted implementation by a single author.
- int_19h 7y ago
  The problem is that you need an XML schema (or at least inline annotations on elements) to designate some value as decimal or float - it's not encoded directly in the data. I think that's what OP really meant.
nicoburns 7y ago
Have you seen RON (Rust Object Notation) https://github.com/ron-rs/ron https://github.com/ron-rs/ron ?
- kstenerud 7y ago
  Yes, but it's too rust-specific. For example, specifying struct names ties the document to a particular implementation, and isn't helpful for a general purpose data representation. Also, it's text only. RON is fine for rust, but not for other languages that might not have its feature set.

pinkfoot 7y ago

ASN.1 ?

kstenerud 7y ago
Too complicated, too many extensions, too bloated on the wire.
- C1sc0cat 7y ago
  What you didn't have fun decoding OSI protocols :-) I was a bit of a wuss and had to use x.409 dumps, my boss could look at the raw data packets and debug problems.
- pinkfoot 7y ago
  PER is bloated?

throwaway3157 7y ago

Cool. When you evaluated all the alternatives, did you stash a pro/con list somewhere on a blog? I’d appreciate reading it

hwbehrens 7y ago
There is a comparison table on the linked GitHub page, with the following formats: Concise (the proposed format), XML, JSON, BSON, CBOR, Messagepack, Cap'n, Protobufs, Flatbuffers, Thrift, and ASN.1. Although not technically a pro/con list, it highlights the evaluation criteria used to compare.
kstenerud 7y ago
I made a short list here [1]. I also kept a design document which explains the choices I made [2]. [1] https://github.com/kstenerud/concise-encoding#comparison-to-other-formats https://github.com/kstenerud/concise-encoding#comparison-to-... [2] https://github.com/kstenerud/concise-encoding/blob/master/design.md https://github.com/kstenerud/concise-encoding/blob/master/de...
- kentonv 7y ago
  Hmm, I think your table is using a different definition of "zero-copy" than is currently common when discussing serialization formats. In your comparison you claim Concise is zero-copy, but it does not appear to be so, at least in the sense that Cap'n Proto and FlatBuffers are. Concise appears to use variable-width integers, tag-value sequences, and 8-byte alignment, all of which make encoded messages unsuitable as in-memory data structures; it looks to me like you have to parse the message into some sort of AST before you could meaningfully operate on the content. "Zero-copy" in the Cap'n Proto / FlatBuffers sense means that there's no need to do any "parsing" because the whole message is efficiently traverseable as-is; e.g. you can mmap() in a very large message and then find and access any one value within it in O(1) time (or perhaps O(log n), if the data structure is nested). This generally requires that primitive values have fixed widths and fields have fixed offsets within their parent object. Digging into your docs it looks like what you really mean by "zero copy" is that individual strings or byte sequences can be used directly without copying them out of the original message buffer. This is a fairly common property of any binary protocol; e.g. Protocol Buffers can do this (albeit without NUL-termination), but in your table you have indicated that it cannot. I would maybe call this "zero-copy strings" to be clearer. On another note, in your table you have indicated that Cap'n Proto doesn't have a "String" type, but this is not true -- Cap'n Proto's "Text" type is a UTF-8 string (and is even NUL-terminated on the wire). (I'm the author of Cap'n Proto.)

carry_bit 7y ago

If JSON was almost right, why not EDN? https://github.com/edn-format/edn https://github.com/edn-format/edn

kstenerud 7y ago
EDN is better than JSON, but it's still text only, and not fully specified.

tasogare 7y ago

The comparison table doesn't seem fair, especially on the XML column. In particular, it is stated that XML doesn't support decimal and float, whereas XSD in fact defines 19 primitive data types including decimal and float. List are also supported implicitly because child nodes of an element are ordered. Moreover, the claim that XML doesn't support metadata is devious because any file can precisely be extended with any additional metadata (possibly in their own namespace), without client breaking if done well. XML strength also lies in its extensibility, and its large number of compliant implementations, which goes way way beyond a single uncompleted implementation by a single author.

int_19h 7y ago
The problem is that you need an XML schema (or at least inline annotations on elements) to designate some value as decimal or float - it's not encoded directly in the data. I think that's what OP really meant.

nicoburns 7y ago

Have you seen RON (Rust Object Notation) https://github.com/ron-rs/ron https://github.com/ron-rs/ron ?

kstenerud 7y ago
Yes, but it's too rust-specific. For example, specifying struct names ties the document to a particular implementation, and isn't helpful for a general purpose data representation. Also, it's text only. RON is fine for rust, but not for other languages that might not have its feature set.