4 ms·

I don't understand why serialization formats that separate structure and content aren't more popular. Imagine a system every message is a UID or DID (https://w

I don't understand why serialization formats that separate structure and content aren't more popular. Imagine a system every message is a UID or DID (https://www.w3.org/TR/did-core/ https://www.w3.org/TR/did-core/) followed by raw binary data. The UID completely describes the shape of the rest of the message. You can also transmit messages to define new UIDs: these messages' UID is a shared global UID that everyone knows about. Once a client learns a UID, messages are about as compact as possible. And the data defining UIDs can be much more descriptive than e.g. property names in JSON. You can send documentation and other excess data when defining the UID, because you don't have to worry about size, because you're only sending the UID once. And UIDs can reference other UIDs to reduce duplication.

mtrovo 5y ago
I think the system OP is describing is a little bit more complex. You're not just describing message types, you also have message templates; a template declares a message type and a set of prefilled fields. You save data by just sending the subset of fields that are actually changing, which is a very good abstraction for market data. The template is hydrated on the protocol parsing layer so your code only has to deal with message types itself.
dboreham 5y ago
This is protocol buffers + a global type registry. I worked on such a system.
- pcarolan 5y ago
  Is it public? Id love to learn more about it.
  - jenny91 5y ago
    If you read the protobuf source, you can see a bunch of places where you can hook in custom type-fetching code, e.g. in the google.protobuf.Any type. After studying it a bit, I'm certain this is how it's used inside Google (might also be mentioned elsewhere). All you'd really need to do is to compile all protos into a repository (you can spit out the binary descriptors from protoc), then fetch those and decode in the client. It'd actually be quite straightforward to set up,
NavinF 5y ago
You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)
- infogulch 5y ago
  Interesting. Ids in particular are described here: https://capnproto.org/language.html#unique-ids https://capnproto.org/language.html#unique-ids I wonder if giving it a name based on the hash of the definition has been explored; like Unison [0] where all code is content addressable, but for just capnproto definitions. Is there a reason not to? [0]: https://www.unisonweb.org https://www.unisonweb.org
  - NavinF 5y ago
    Capnp uses the name of your message, but not its full definition because that would make it impossible to extend protocols in a backwards compatible way. Without the ability to add new fields, making changes to your protocol would be impossible in large orgs.
- boxfire 5y ago
  MD5 is a 128 bit random number no one would ever have thought would collide. 64 bits is peanuts especially when message types are being defined dynamically
  - deleted 5y ago
    [deleted]
  - remram 5y ago
    MD5 is safe against unintentional collisions.
  - NavinF 5y ago

mtrovo 5y ago

I think the system OP is describing is a little bit more complex. You're not just describing message types, you also have message templates; a template declares a message type and a set of prefilled fields. You save data by just sending the subset of fields that are actually changing, which is a very good abstraction for market data. The template is hydrated on the protocol parsing layer so your code only has to deal with message types itself.

dboreham 5y ago

This is protocol buffers + a global type registry. I worked on such a system.

pcarolan 5y ago
Is it public? Id love to learn more about it.
- jenny91 5y ago
  If you read the protobuf source, you can see a bunch of places where you can hook in custom type-fetching code, e.g. in the google.protobuf.Any type. After studying it a bit, I'm certain this is how it's used inside Google (might also be mentioned elsewhere). All you'd really need to do is to compile all protos into a repository (you can spit out the binary descriptors from protoc), then fetch those and decode in the client. It'd actually be quite straightforward to set up,

NavinF 5y ago

You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)

infogulch 5y ago
Interesting. Ids in particular are described here: https://capnproto.org/language.html#unique-ids https://capnproto.org/language.html#unique-ids I wonder if giving it a name based on the hash of the definition has been explored; like Unison [0] where all code is content addressable, but for just capnproto definitions. Is there a reason not to? [0]: https://www.unisonweb.org https://www.unisonweb.org
- NavinF 5y ago
  Capnp uses the name of your message, but not its full definition because that would make it impossible to extend protocols in a backwards compatible way. Without the ability to add new fields, making changes to your protocol would be impossible in large orgs.
boxfire 5y ago
MD5 is a 128 bit random number no one would ever have thought would collide. 64 bits is peanuts especially when message types are being defined dynamically
- deleted 5y ago
  [deleted]
- remram 5y ago
  MD5 is safe against unintentional collisions.
- NavinF 5y ago