4 ms·

Not widely used but I like Typical's approach https://github.com/stepchowfun/typical https://github.com/stepchowfun/typical > Typical offers a new solution ("

by jitl 1y ago

Not widely used but I like Typical's approach https://github.com/stepchowfun/typical https://github.com/stepchowfun/typical > Typical offers a new solution ("asymmetric" fields) to the classic problem of how to safely add or remove fields in record types without breaking compatibility. The concept of asymmetric fields also solves the dual problem of how to preserve compatibility when adding or removing cases in sum types.

cornstalks 1y ago
I've never heard of Typical but the fact they didn't repeat protobuf's sin regarding varint encoding (or use leb128 encoding...) makes me very interested! Thank you for sharing, I'm going to have to give it a spin.
- zigzag312 1y ago
  It looks similar to how vint64 lib encodes varints. Total length of varint can be determined via the first byte alone.
  - haberman 1y ago
    I advocated for PrefixVarint (which seems equivalent to vint64 ) for WebAssembly, but it was decided against, in favor of LEB128: https://github.com/WebAssembly/design/issues/601 https://github.com/WebAssembly/design/issues/601 The recent CREL format for ELF also uses the more established LEB128: https://news.ycombinator.com/item?id=41222021 https://news.ycombinator.com/item?id=41222021 At this point I don't feel like I have a clear opinion about whether PrefixVarint is worth it, compared with LEB128.
    - zigzag312 1y ago
      Just remember that XML was more established than JSON for a long time.
    - kannanvijayan 1y ago
      Varint encoding is something I've peeked at in various contexts. My personal bias is towards the prefix-style, as it feels faster to decode and the segregation of the meta-data from the payload data is nice. But, the thing that tends to tip the scales is the fact that in almost all real world cases, small numbers dominate - as the github thread you linked relates in a comment. The LEB128 fast-path is a single conditional with no data-dependencies: if ! (x & 0x80) { x } Modern CPUs will characterize that branch really well and you'll pay almost zero cost for the fastpath which also happens to be the dominant path. It's hard to beat.
      - yencabulator 1y ago
        SQLite format equivalent: if x <= 240 { x } while strictly improving all other aspects (at least IMHO) https://sqlite.org/src4/doc/trunk/www/varint.wiki https://sqlite.org/src4/doc/trunk/www/varint.wiki
zigzag312 1y ago
This actually looks quite interesting.
summerlight 1y ago
This seems interesting. Still not sure if `required` is a good thing to have (for persistent data like log you cannot really guarantee some field's presence without schema versioning baked into the file itself) but for an intermediate wire use cases, this will help.
rkagerer 1y ago
More direct link to the juicy bit: https://github.com/stepchowfun/typical?tab=readme-ov-file#asymmetric-fields-can-safely-be-promoted-to-required-and-vice-versa https://github.com/stepchowfun/typical?tab=readme-ov-file#as... An asymmetric field in a struct is considered required for the writer, but optional for the reader.
- sdenton4 1y ago
  That's a nice idea... But I believe the design direction of proto buffers was to make everything `optional`, because `required` tends to bite you later when you realize it should actually be optional.
  - bilkow 1y ago
    My understanding is that asymmetric fields provide a migration path in case that happens, as stated in the docs: > Unlike optional fields, an asymmetric field can safely be promoted to required and vice versa. > [...] > Suppose we now want to remove a required field. It may be unsafe to delete the field directly, since then clients might stop setting it before servers can handle its absence. But we can demote it to asymmetric, which forces servers to consider it optional and handle its potential absence, even though clients are still required to set it. Once that change has been rolled out (at least to servers), we can confidently delete the field (or demote it to optional), as the servers no longer rely on it.
    - yencabulator 1y ago
      > My understanding is that asymmetric fields provide a migration path in case that happens, as stated in the docs: If you can assume you can churn a generation of fresh data soonish, and never again read the old data. For RPC sure, but someone like Google has petabytes of stored protobufs, so they don't pretend they can upgrade all the writers.
    - sdenton4 1y ago
      ....or we can just say that everything is optional always, and leave it to the servers instead of the protocol to handle irregularities.
atombender 1y ago
I'm really hoping Typical will catch on, as I quite like the design. One important gap right now is the lack of Go and Python support.
sevensor 1y ago
Seems like a lot of effort to avoid adding a message version field. I’m not a web guy, so maybe I’m missing the point here, but I always embed a schema version field in my data.
- vouwfietsman 1y ago
  I get that. The point is that its hard to prevent asymmetry in message versions if you are working with many communicating systems. Lets say four services inter-communicate with some protocol, it is extremely annoying to impose a deployment order where the producer of a message type is the last to upgrade the message schema, as this causes unnecessary dependencies between the release trains of these services. At the same time, one cannot simply say: "I don't know this message version, I will disregard it" because in live systems this will mean the systems go out of sync, data is lost, stuff breaks, etc. There's probably more issues I haven't mentioned, but long story short: in live, interconnected systems, it becomes important to have intelligent message versioning, i.e: a version number is not enough.
  - sevensor 1y ago
    I think I see what you’re getting at? My mental model is client and server, but you’re implying a more complex topology where no one service is uniquely a server or a client. You’d like to insert a new version at an arbitrary position in the graph without worrying about dependencies or the operational complexity of doing a phased deployment. The result is that you try to maintain a principled, constructive ambiguity around the message schema, hence asymmetrical fields? I guess I’m still unconvinced and I may have started the argument wrong, but I can see a reasonable person doing it that way.
    - vouwfietsman 1y ago
      Yes thats a big part, but even bigger is just the alignment of teams. Imagine team A building feature XYZ Team B is building TUV one of those features in each team deals with messages, the others are unrelated. At some point in time, both teams have to deploy. If you have to sync them up just to get the protocol to work, thats an extra complexity in the already complex work of the teams. If you can ignore this, great! It becomes even more complex with rolling updates though: not all deployments of a service will have the new code immediately, because you want multiple to be online to scale on demand. This creates an immediate necessary ambiguity in the qeustion: "which version does this service accept?" because its not about the service anymore, but about the deployments.

by jitl 1y ago

cornstalks 1y ago

I've never heard of Typical but the fact they didn't repeat protobuf's sin regarding varint encoding (or use leb128 encoding...) makes me very interested! Thank you for sharing, I'm going to have to give it a spin.

zigzag312 1y ago
It looks similar to how vint64 lib encodes varints. Total length of varint can be determined via the first byte alone.
- haberman 1y ago
  I advocated for PrefixVarint (which seems equivalent to vint64 ) for WebAssembly, but it was decided against, in favor of LEB128: https://github.com/WebAssembly/design/issues/601 https://github.com/WebAssembly/design/issues/601 The recent CREL format for ELF also uses the more established LEB128: https://news.ycombinator.com/item?id=41222021 https://news.ycombinator.com/item?id=41222021 At this point I don't feel like I have a clear opinion about whether PrefixVarint is worth it, compared with LEB128.
  - zigzag312 1y ago
    Just remember that XML was more established than JSON for a long time.
  - kannanvijayan 1y ago
    Varint encoding is something I've peeked at in various contexts. My personal bias is towards the prefix-style, as it feels faster to decode and the segregation of the meta-data from the payload data is nice. But, the thing that tends to tip the scales is the fact that in almost all real world cases, small numbers dominate - as the github thread you linked relates in a comment. The LEB128 fast-path is a single conditional with no data-dependencies: if ! (x & 0x80) { x } Modern CPUs will characterize that branch really well and you'll pay almost zero cost for the fastpath which also happens to be the dominant path. It's hard to beat.
    - yencabulator 1y ago
      SQLite format equivalent: if x <= 240 { x } while strictly improving all other aspects (at least IMHO) https://sqlite.org/src4/doc/trunk/www/varint.wiki https://sqlite.org/src4/doc/trunk/www/varint.wiki

zigzag312 1y ago

This actually looks quite interesting.

summerlight 1y ago

This seems interesting. Still not sure if `required` is a good thing to have (for persistent data like log you cannot really guarantee some field's presence without schema versioning baked into the file itself) but for an intermediate wire use cases, this will help.

rkagerer 1y ago

More direct link to the juicy bit: https://github.com/stepchowfun/typical?tab=readme-ov-file#asymmetric-fields-can-safely-be-promoted-to-required-and-vice-versa https://github.com/stepchowfun/typical?tab=readme-ov-file#as... An asymmetric field in a struct is considered required for the writer, but optional for the reader.

sdenton4 1y ago
That's a nice idea... But I believe the design direction of proto buffers was to make everything `optional`, because `required` tends to bite you later when you realize it should actually be optional.
- bilkow 1y ago
  My understanding is that asymmetric fields provide a migration path in case that happens, as stated in the docs: > Unlike optional fields, an asymmetric field can safely be promoted to required and vice versa. > [...] > Suppose we now want to remove a required field. It may be unsafe to delete the field directly, since then clients might stop setting it before servers can handle its absence. But we can demote it to asymmetric, which forces servers to consider it optional and handle its potential absence, even though clients are still required to set it. Once that change has been rolled out (at least to servers), we can confidently delete the field (or demote it to optional), as the servers no longer rely on it.
  - yencabulator 1y ago
    > My understanding is that asymmetric fields provide a migration path in case that happens, as stated in the docs: If you can assume you can churn a generation of fresh data soonish, and never again read the old data. For RPC sure, but someone like Google has petabytes of stored protobufs, so they don't pretend they can upgrade all the writers.
  - sdenton4 1y ago
    ....or we can just say that everything is optional always, and leave it to the servers instead of the protocol to handle irregularities.

atombender 1y ago

I'm really hoping Typical will catch on, as I quite like the design. One important gap right now is the lack of Go and Python support.

sevensor 1y ago

Seems like a lot of effort to avoid adding a message version field. I’m not a web guy, so maybe I’m missing the point here, but I always embed a schema version field in my data.

vouwfietsman 1y ago
I get that. The point is that its hard to prevent asymmetry in message versions if you are working with many communicating systems. Lets say four services inter-communicate with some protocol, it is extremely annoying to impose a deployment order where the producer of a message type is the last to upgrade the message schema, as this causes unnecessary dependencies between the release trains of these services. At the same time, one cannot simply say: "I don't know this message version, I will disregard it" because in live systems this will mean the systems go out of sync, data is lost, stuff breaks, etc. There's probably more issues I haven't mentioned, but long story short: in live, interconnected systems, it becomes important to have intelligent message versioning, i.e: a version number is not enough.
- sevensor 1y ago
  I think I see what you’re getting at? My mental model is client and server, but you’re implying a more complex topology where no one service is uniquely a server or a client. You’d like to insert a new version at an arbitrary position in the graph without worrying about dependencies or the operational complexity of doing a phased deployment. The result is that you try to maintain a principled, constructive ambiguity around the message schema, hence asymmetrical fields? I guess I’m still unconvinced and I may have started the argument wrong, but I can see a reasonable person doing it that way.
  - vouwfietsman 1y ago
    Yes thats a big part, but even bigger is just the alignment of teams. Imagine team A building feature XYZ Team B is building TUV one of those features in each team deals with messages, the others are unrelated. At some point in time, both teams have to deploy. If you have to sync them up just to get the protocol to work, thats an extra complexity in the already complex work of the teams. If you can ignore this, great! It becomes even more complex with rolling updates though: not all deployments of a service will have the new code immediately, because you want multiple to be online to scale on demand. This creates an immediate necessary ambiguity in the qeustion: "which version does this service accept?" because its not about the service anymore, but about the deployments.