3 ms·

Obligatory Cap'n Proto[1] reference whenever serialization format discussions crop up. [1] https://capnproto.org/index.html https://capnproto.org/index.html

mikepurvis 7y ago
Capnproto is schema based rather than self-describing. So it's related, and it does compete with formats like json and msgpack, but there isn't total overlap in the applications.
- twic 7y ago
  I would be nice to have a taxonomy, or at least a bestiary, of serialisation formats. Tentatively: Schema-driven no-compromise fast compact binary formats with no cross-version compatibility: Cap'n Proto, FlatBuffers, SBE, ASN.1 PER, XDR, OMG CDR. Schema-driven binary formats which allow some cross-version compatibility: Protocol Buffers, Thrift. Self-describing binary formats: MessagePack, CBOR, BJSON, Bencode, ASN.1 BER, Avro (?), Fast Infoset, AMF3. Self-describing textual formats: JSON, XML, YAML, TOML. I'm using "self-describing" here to mean simply that you can recover the structure of the encoded data without a separate schema, rather than that you can attach any semantic meaning to it.
  - kentonv 7y ago
    > no cross-version compatibility: Cap'n Proto, FlatBuffers This is incorrect: Cap'n Proto absolutely allows cross-version compatibility, using roughly the same semantics as Protobuf. I believe FlatBuffers does too. (I'm unsure about the rest, haven't studied them in a while.) > I'm using "self-describing" here to mean simply that you can recover the structure of the encoded data without a separate schema, rather than that you can attach any semantic meaning to it. Protobuf, Cap'n Proto, and probably several of the other binary formats can parse data into a message tree without the help of a schema, but all the fields will be labeled numerically. MessagePack is only considered "self-describing" in comparison because in encodes human-readable field names on the wire.

mikepurvis 7y ago

Capnproto is schema based rather than self-describing. So it's related, and it does compete with formats like json and msgpack, but there isn't total overlap in the applications.

twic 7y ago
I would be nice to have a taxonomy, or at least a bestiary, of serialisation formats. Tentatively: Schema-driven no-compromise fast compact binary formats with no cross-version compatibility: Cap'n Proto, FlatBuffers, SBE, ASN.1 PER, XDR, OMG CDR. Schema-driven binary formats which allow some cross-version compatibility: Protocol Buffers, Thrift. Self-describing binary formats: MessagePack, CBOR, BJSON, Bencode, ASN.1 BER, Avro (?), Fast Infoset, AMF3. Self-describing textual formats: JSON, XML, YAML, TOML. I'm using "self-describing" here to mean simply that you can recover the structure of the encoded data without a separate schema, rather than that you can attach any semantic meaning to it.
- kentonv 7y ago
  > no cross-version compatibility: Cap'n Proto, FlatBuffers This is incorrect: Cap'n Proto absolutely allows cross-version compatibility, using roughly the same semantics as Protobuf. I believe FlatBuffers does too. (I'm unsure about the rest, haven't studied them in a while.) > I'm using "self-describing" here to mean simply that you can recover the structure of the encoded data without a separate schema, rather than that you can attach any semantic meaning to it. Protobuf, Cap'n Proto, and probably several of the other binary formats can parse data into a message tree without the help of a schema, but all the fields will be labeled numerically. MessagePack is only considered "self-describing" in comparison because in encodes human-readable field names on the wire.