7 ms·

Typical: Data interchange with algebraic data types

• tiziano88 3y ago
  Brilliant, I have been thinking of doing exactly this for a while now, glad I waited for someone else to do it in a better way :)
• habitue 3y ago
  This seems like the thing I wish protocol buffers were, but learning from them and removing the warts. Very promising. Also, implementing in rust and typescript as the first languages was a smart choice
  • revskill 3y ago
    Protocol buffer lacks of good documentation for beginners which fits in a README.md. It's not a community project i guess. Basically enterprisey toolings bored me to death. Nowadays i trust toolings whose documentation could fit in a README file. It means it's concise, simplicity.
    • kubanczyk 3y ago
      This comes close https://protobuf.dev/programming-guides/encoding/ https://protobuf.dev/programming-guides/encoding/ It's long, but dev-friendly. Once I got through, I could understand 99% of benefits and limitations of protobufs.
  • anitil 3y ago
    I don't know a lot about protocol buffers. What are the warts? I think they don't have a canonical format, is that one of them?
• klysm 3y ago
  Looks phenomenal, especially regarding robustness around union types
• bombolo 3y ago
  It seems we're back to corba
  • xupybd 3y ago
    IMO Corba was still better than SOAP.
• munificent 3y ago
  Asymmetric fields is a really clever idea.
• xupybd 3y ago
  I'd love to use this if it had .net support. We use Nats to communicate between services and use JSON for a lack of better options. I've been looking for something more efficient and strict. This looks like it would be a good match for F#s types.
• pickledish 3y ago
  > Note: An optional field in a choice isn't simply a field with an option type/nullable type. The word "optional" here means that readers can ignore it and use a fallback instead, not that its payload might be missing Hmm, this is new to me -- is this a Typical concept, or something more general around protobuf-enums that I just haven't run into before? (oh, and besides my question, strong agreement with everyone else here, cool lib and great documentation!)
  • stepchowfun 3y ago
    This is a Typical concept. I haven't seen this approach to optionality for sum types in any other serialization framework. Programmers have good intuition for what it means for a field in a struct to be optional: it's either there or it's not. But for a sum type, what does it mean for a case to be "missing"? It's not quite as obvious, but it helps to think about what responsibilities are placed on writers vs. readers: optionality for a struct relaxes the burden on writers (they don't have to set the field), whereas for a sum type the burden is relaxed on readers (they don't have to handle the case).
• parhamn 3y ago
  Curious how this will look when they get to implementations with less expressive type systems. Typescript & Rust are particularly good. Making a usable library for this in golang won't be easy. And now that I think about it, Protobuf/Thrift/etc type tools are heavily constrained by finding lowest-common-demoninator of language features to allow for cross serialization. Maybe in the next generation of these tools, languages like golang don't get a seat at the table for the sake of progress -- I could be fine with it.
  • stepchowfun 3y ago
    You're exactly right about other frameworks appealing to the lowest common denominator, whereas Typical isn't willing to make such compromises. Languages without proper sum types are at a disadvantage here, but it's possible to encode sum types with exhaustive pattern matching in such languages using the visitor pattern. That approach requires some ergonomic sacrifices (e.g., having to use a reified eliminator rather than the built-in `switch` statement), and people using those languages may prefer convenience over strong guarantees. It's an unfortunate impedance mismatch.
  • jcparkyn 3y ago
    I'd imagine that most people voluntarily using go or similar languages wouldn't be too bothered by just having all the checks occur at runtime in the generated code, rather than encoding them in the type system. Sum types are still awkward, but most languages can at least approximate them, minus some compile-time checks.
• stepchowfun 3y ago
  Typical creator here. I'm pleasantly surprised to find this on HN today! Happy to answer any questions about it.
  • andyferris 3y ago
    I was wondering if you've considered having an alternative, human readable encoding (either your own syntax or a JSON-based schema)? I find it quite useful to be able to inspect data by eye and even hand-editing payloads occassionally, and having a standard syntax for doing so would be nice. (More generally, it's a shame JSON doesn't support sum types "natively" and I think a human readable format with Typical's data model would be really cool).
    • stepchowfun 3y ago
      It's a good question! The binary format is completely inscrutable to human eyes and is not designed for manual inspection/editing. However: 1) For Rust, the generated types implement the `Debug` trait, so they can be printed in a textual format that Rust programmers are accustomed to reading. 2) For JavaScript/TypeScript, deserialized messages are simple passive data objects that can be logged or inspected in the developer console. So it's easy to log/inspect deserialized messages in a human-readable format, but there's currently no way to read/edit encoded messages directly. In the future, we may add an option to encode messages as JSON which would match the representation currently used for decoded messages in JavaScript/TypeScript, with sums being encoded as tagged unions.
      • andyferris 3y ago
        Thanks. That's a good answer.
    • magicalhippo 3y ago
      > it's a shame JSON doesn't support sum types "natively" You can describe it using JSON Schema though[1], using "oneOf" and "const". Though I prefer the more explicit way of using "oneOf" combined with "required" to select one of a number of keys[2]. [1]: https://www.jsonschemavalidator.net/s/6SCuYNBe https://www.jsonschemavalidator.net/s/6SCuYNBe [2]: https://www.jsonschemavalidator.net/s/tNnQmsTd https://www.jsonschemavalidator.net/s/tNnQmsTd
• iso8859-1 3y ago
  It seems that there are two approaches to schema encoding: * Writing OpenAPI/Avro and then generating deserializing/serializing from that (like e.g. Avro[0] or Tie[1] or Typical) * Writing the schema using an in-language DSL (e.g. Autodocodec[2]) If I have a single language codebase, why should I prefer the first approach? You can always make your in-language DSL serialize out to a dedicated language at a later point. Typical isn't focused on JSON, so it doesn't seem like it is optimized for web. Not doing web makes it more likely that you don't need multiple language support. You can limit the metaprogramming also: You don't strictly need GHC.Generics for the in-language DSL. But if you're generating code, it's always going to be opaque and hard to debug. If you keep the DSL in-language, you don't need to generate stubs since you can use the languages own type system to enforce the mapping to the native records[2]. I have heard the argument that everything should be 'documentation first', which was given as an argument for using Tie. But I don't see why an in-language DSL can't provide enough detail. There is so much manually written OpenAPI out there, any of these approaches is vastly better than that. I have been reading Designing Data Intensive Applications by Martin Kleppmann but it doesn't cover this trade-off. Which makes sense, since it isn't really a book on programming using DSLs. [0]: https://hackage.haskell.org/package/avro#generating-code-from-avro-schema https://hackage.haskell.org/package/avro#generating-code-fro... [1]: https://github.com/scarf-sh/tie https://github.com/scarf-sh/tie [2]: https://github.com/NorfairKing/autodocodec#fully-featured-example https://github.com/NorfairKing/autodocodec#fully-featured-ex...
  • stepchowfun 3y ago
    > If I have a single language codebase, why should I prefer the first approach? Probably the most compelling reason is that a single language codebase might not be a single language codebase forever. But, as you suggested, the switch to a language-agnostic framework can be deferred until it becomes necessary. However, there's a reason to use Typical specifically: asymmetric fields. This feature allows you to change your schema over time without breaking compatibility and without sacrificing type safety. If you ever expect to have newer versions of the codebase reading messages that were generated by older versions of the codebase (or vice versa), this is a concern that will need to be addressed. This can happen when you have a system that isn't deployed atomically (e.g., microservices, web/mobile applications, etc.) or when you have persisted messages that can outlive a single version of the codebase (e.g., files or database records). An embedded DSL could in principle provide asymmetric fields, but I'm not aware of any that do. > Typical isn't focused on JSON, so it doesn't seem like it is optimized for web. It just makes different trade-offs than most web-based systems, but that doesn't make it unsuitable for web use. We have comprehensive integration tests that run in the browser. Deserialized messages are simple passive data objects [1] that can be logged or inspected in the developer console. [1] https://en.wikipedia.org/wiki/Passive_data_structure https://en.wikipedia.org/wiki/Passive_data_structure
• naasking 3y ago
  Nice that algebraic types are getting more love. Would be nice if these could be imported into existing systems, like Cap 'n Proto.
• spenczar5 3y ago
  > Typical offers a new solution ("asymmetric" fields) to the classic problem of how to safely add or remove fields in record types without breaking compatibility. The concept of asymmetric fields also solves the dual problem of how to preserve compatibility when adding or removing cases in sum types. This is indeed an elegant solution, although I am not so sure about its novelty. It seems very similar to how Avro achieves forward and backward compatibility. I wonder how the two strategies differ?
  • stepchowfun 3y ago
    Avro has no equivalent of Typical's asymmetric fields. In Avro: 1. Record types can have optional (but not asymmetric) fields, just like in most IDLs. Avro implements this by taking the union of the field type with a special `null` type, but in practice it's equivalent to having optional fields. 2. Avro doesn't support proper sum types, but it has two approximations of them: unions (but not tagged unions) and enums. Unions have no support for adding/removing new cases safely. Enums can have a default value that is used if the case is not recognized. From the "Typical perspective", both of these are problematic: 1. If you are trying to introduce a required field in Avro, you first introduce it as optional, and then at some point in the future when you have manually confirmed that the field is always being set by every writer, you can promote the field to required. Typical's asymmetric fields offload the burden of having to do that manual confirmation onto the type checker. 2. For unions, Avro offers no equivalent of optional or asymmetric cases. If you add a new case, you better not use it until all readers can handle it, and the type checker won't enforce that. For enums, the solution of having default values is unsatisfactory, because not every type has a suitable default. In practice, this usually means adding a catch-all default that is only used to signal an unrecognized input, but then what is a reader supposed to do with that? With Typical, if a writer introduces a new optional or asymmetric case, it must then provide an appropriate fallback to use when the new case isn't recognized by readers. For example, if a new specific type of error is introduced (e.g., with a stack trace), the fallback might be a more general type of error (e.g., with only an error message) that readers already know how to handle. If the new case is asymmetric, then you know readers can handle it, so once its rolled out you know it's safe to subsequently promote the case to required (so that writers no longer need to provide a fallback). Here I've only discussed the challenges with adding new required fields/cases, but you run into similar trouble when removing them. This section of the README discusses all the pitfalls: https://github.com/stepchowfun/typical#required-optional-and-asymmetric-fields https://github.com/stepchowfun/typical#required-optional-and...
• padator 3y ago
  What is the difference with ATD: https://atd.readthedocs.io/en/latest/atd-project.html https://atd.readthedocs.io/en/latest/atd-project.html which also was designed with algebraic data types in mind?
  • stepchowfun 3y ago
    I can't speak with authority about ATD, but the following might be helpful: Aside from algebraic data types, the big selling point of Typical is asymmetric fields. That's the crucial feature that distinguishes Typical from every other framework. Without asymmetric fields, there is no safe way to introduce or retire required fields. People using other frameworks fear required fields (rightly so), whereas Typical gives you the tools to embrace them.
• Lacaranian 3y ago
  I love everything about this! I think a lot of code could benefit from restructuring via ADTs, and ser/deser is an important piece of that story. But I suppose I do have one nitpick. Using a fallback for asymmetric fields in sum types seems off to me, albeit pragmatic. If the asymmetric fields for product types use an Option<T>, and Option is basically a sum of a T and a Unit, a close dual is a product (struct/tuple) of a T and the dual of Unit (a Nothing type, eg. one with no instantiable values, such as an empty enum). I think this would provide similar safety guarantees, as a writer couldn't produce a value of an added asymmetric sum type variant, but a reader could write handling for it (including all subfields besides the Nothing typed one)?
  • stepchowfun 3y ago
    You're considering an alternative behavior for asymmetric fields in choices, but you need to consider the behavior of optional fields in choices too. In particular, the following duality is the lynchpin that ties everything together: "asymmetric" behaves like optional for struct readers and choice writers, but required for struct writers and choice readers. From that duality, the behavior of asymmetric fields is completely determined by the behavior of optional fields. It isn't up to us to decide arbitrarily. So the question becomes: what is the expected behavior of optional choice fields? Along the lines you proposed, one could try to make optional choice fields behave as if they had an uninhabited type, so that writers would be unable to instantiate them—then you get exactly the behavior you described for asymmetric choice fields. Optional fields are symmetric, so both readers and writers would treat them as the empty type. This satisfies the safety requirement, but only in a degenerate way: optional fields would then be completely useless. So this is not the way to go. It's important to take a step back and consider what "optional" ought to mean: optionality for a struct relaxes the burden on writers (they don't have to set the field), whereas for a choice the burden is relaxed on readers (they don't have to handle the field). So how do you allow readers to ignore a choice field? Not by refusing to construct it (which would make it useless), but rather by providing a fallback. So the insight is not to think of optional as a type operator (1 + T) that should be dualised in some way (0 * T), but rather to think about the requirements imposed on writers and readers. You're right to note the duality between sums and products and initial and terminal objects, and indeed category theory had a strong influence on Typical's design.
• Kinrany 3y ago
  Asymmetric fields are great. Shouldn't they be the default?
  • stepchowfun 3y ago
    Asymmetric fields are in a temporary transition state to/from required. So it would seem a bit odd to me for the transition state to be the default. However, I think I see your reasoning: you don't want to accidentally introduce a required field without first making it asymmetric, and having asymmetric as the default would effectively prevent that. So I can see the appeal of both designs!