4 ms·

I regret to say that every type level Gordian knot that I have ever been exposed to came from attempts to do this. I think a lot of the problem is that many of

by taeric 2y ago

I regret to say that every type level Gordian knot that I have ever been exposed to came from attempts to do this. I think a lot of the problem is that many of the constraints and acceptable values for data are not determined until well after first deployment. The "knot" comes in when you are explicitly changing data and code to add a feature that you didn't anticipate at the start. This is a lot like schemaless databases. The flexibility of not having to fully specify the full schema does not mean that you don't benefit from specifying parts of it? Indeed, if you are indexing things, you have to specify those parts. But it is hard to argue with a straight face that schemaless tools don't have some strong benefits.

thesuperbigfrog 2y ago
>> This is a lot like schemaless databases. The flexibility of not having to fully specify the full schema does not mean that you don't benefit from specifying parts of it? Indeed, if you are indexing things, you have to specify those parts. But it is hard to argue with a straight face that schemaless tools don't have some strong benefits. This is very similar to what Rich Hickey argued in "Simplicity Matters": https://youtu.be/rI8tNMsozo0?si=xTkpsLYTYh0jA5lB https://youtu.be/rI8tNMsozo0?si=xTkpsLYTYh0jA5lB Basically when you use a hash / JavaScript-style object rather than a fixed struct definition for your data (which amounts to a schema- less database in aggregate) Easy addition of new properties Co-existence of different versions of objects (newer objects will typically have more properties) Introspection of properties and validation rules / reflection
- wwweston 2y ago
  Sometimes I’ve wondered if versioned types might be a help in bringing coexistence to a typed setup, and if this is part of where the benefits of a service oriented architecture come from. But given how versioning snarls can play out in dependency management I expect this idea has some tradeoffs in the best case.
  - taeric 2y ago
    I'm curious what you have in mind that doesn't boil down to "duck typing?"
    - wwweston 2y ago
      Not the most elegant mechanism, but it should be accessible to imagine an implementation via inheritance hierarchies even for a static manifestly typed language. Class "Person4.1" inherits from class "Person4" inherits from class "Person2" inherits from "Person". Probably there's a better way (and one could argue that a design where knowing the version of the class/object matters isn't a well-encapsulated design, which could have something to do with why duck typing often works out better than expected in OO systems).
      - taeric 2y ago
        I think my question is what would you be doing here that doesn't boil down to "has these properties/methods, I'll accept"? I think this is a lot easier if you exclude the "methods" in my quote there. Since those almost certainly have the same general contract that would spread across things.
  - satvikpendem 2y ago
    You might be interested in the Unison language then.
- skybrian 2y ago
  Protobufs are an interesting hybrid. They assume a common, linear history of schema versions, where all data generators had access to an arbitrary version in that history. The schemas define field ids and types, but not which combinations of fields can show up. If you can upgrade all the data then you don’t need to worry about versioning, Or you can do it like HTTP headers and hope for the best.
  - taeric 2y ago
    Protobufs are also a fun place to look for how much debate people will get into regarding required versus optional fields. Your post is taking an implicit "everything is optional" view. But, it does allow you to be stricter. Common Lisp Object System also touched on all of these ideas years ago.
    - smackmybishop 2y ago
      Everything has been `optional` for about a decade. > The option to set a field to required is absent in proto3 and strongly discouraged in proto2. https://protobuf.dev/overview/#syntax https://protobuf.dev/overview/#syntax
      - taeric 2y ago
        Ah, I dropped off protobuf a while back. I definitely remember a lot of uproar about it, at the time.
- diffxx 2y ago
  The first two properties can be handled by adding optional fields to your struct. The third can be done at compile time rather than runtime if you define a strict class hierarchy. I think really the problem is that many languages make struct definition needlessly verbose so it feels convenient to just use a map. The price of using schemaless includes runtime performance as well as the compiler being unable to warn you about easy (for a compiler) to detect and report data mismatch errors. I have spent a lot of time using lua the way I think many people use clojure. It is quite seductive and does feel very productive at first, but eventually I find that for a sufficiently complex program, I want the compiler to validate my data usage for me.
nucleardog 2y ago
> This is a lot like schemaless databases. The flexibility of not having to fully specify the full schema does not mean that you don't benefit from specifying parts of it? Indeed, if you are indexing things, you have to specify those parts. But it is hard to argue with a straight face that schemaless tools don't have some strong benefits. Let me put on my straightest face. Schemaless tools have no benefits.* As soon as you start _doing_ anything with your data, it has a schema. You may not have declared it to your data store, to a specific piece of code, etc, but your application logic almost definitely makes assumptions about the shape of the data--what fields are and aren't present, the types of values in those fields, and what those values mean. Every piece of your system must either accept undefined behaviour in the face of data which doesn't fit this implicit schema (there are very few non-trivial problems where you can define a meaningful, valid output for all possible inputs), or essentially treat the schemaless parts as hostile and assert its assumptions before every action. The only thing you've done by going "schemaless" is distributing your schema across your systems and pushed the schema validation out to each thing interacting with the schemaless components instead of centralizing it. * Yes, literally storing raw, unstructured data. But as soon as you need to _do_ anything with the data besides regurgitate it you're back to it having some sort of schema.
- taeric 2y ago
  So you have mainly straw-manned on the word "schemaless" to make an argument. This is akin to attacking dynamic typing by pointing out that most things can be type inferred by the code. That is, you are not wrong in that most data stored is structured in some way; but you are ignoring a ton of the benefit of the schemaless world. As I pointed out, there are benefits to specifying schema. In parts, you /have/ to do so. Specifically if you want your DB to do any work on the data. Indexing/filtering and the like. Now, you do hit largely on the benefits of the "schemaless" world. And that is that it is not undefined how it will store whatever schema you throw at it. You will have restrictions on keys, but otherwise it will store whatever you throw at it. At large, this means you can skip out on a lot of the formality of specifying all schema shapes during development and can take a much more flexible approach in your code than you are forced to if the data layer can't work with you.

by taeric 2y ago

thesuperbigfrog 2y ago

>> This is a lot like schemaless databases. The flexibility of not having to fully specify the full schema does not mean that you don't benefit from specifying parts of it? Indeed, if you are indexing things, you have to specify those parts. But it is hard to argue with a straight face that schemaless tools don't have some strong benefits. This is very similar to what Rich Hickey argued in "Simplicity Matters": https://youtu.be/rI8tNMsozo0?si=xTkpsLYTYh0jA5lB https://youtu.be/rI8tNMsozo0?si=xTkpsLYTYh0jA5lB Basically when you use a hash / JavaScript-style object rather than a fixed struct definition for your data (which amounts to a schema- less database in aggregate) Easy addition of new properties Co-existence of different versions of objects (newer objects will typically have more properties) Introspection of properties and validation rules / reflection

wwweston 2y ago
Sometimes I’ve wondered if versioned types might be a help in bringing coexistence to a typed setup, and if this is part of where the benefits of a service oriented architecture come from. But given how versioning snarls can play out in dependency management I expect this idea has some tradeoffs in the best case.
- taeric 2y ago
  I'm curious what you have in mind that doesn't boil down to "duck typing?"
  - wwweston 2y ago
    Not the most elegant mechanism, but it should be accessible to imagine an implementation via inheritance hierarchies even for a static manifestly typed language. Class "Person4.1" inherits from class "Person4" inherits from class "Person2" inherits from "Person". Probably there's a better way (and one could argue that a design where knowing the version of the class/object matters isn't a well-encapsulated design, which could have something to do with why duck typing often works out better than expected in OO systems).
    - taeric 2y ago
      I think my question is what would you be doing here that doesn't boil down to "has these properties/methods, I'll accept"? I think this is a lot easier if you exclude the "methods" in my quote there. Since those almost certainly have the same general contract that would spread across things.
- satvikpendem 2y ago
  You might be interested in the Unison language then.
skybrian 2y ago
Protobufs are an interesting hybrid. They assume a common, linear history of schema versions, where all data generators had access to an arbitrary version in that history. The schemas define field ids and types, but not which combinations of fields can show up. If you can upgrade all the data then you don’t need to worry about versioning, Or you can do it like HTTP headers and hope for the best.
- taeric 2y ago
  Protobufs are also a fun place to look for how much debate people will get into regarding required versus optional fields. Your post is taking an implicit "everything is optional" view. But, it does allow you to be stricter. Common Lisp Object System also touched on all of these ideas years ago.
  - smackmybishop 2y ago
    Everything has been `optional` for about a decade. > The option to set a field to required is absent in proto3 and strongly discouraged in proto2. https://protobuf.dev/overview/#syntax https://protobuf.dev/overview/#syntax
    - taeric 2y ago
      Ah, I dropped off protobuf a while back. I definitely remember a lot of uproar about it, at the time.
diffxx 2y ago
The first two properties can be handled by adding optional fields to your struct. The third can be done at compile time rather than runtime if you define a strict class hierarchy. I think really the problem is that many languages make struct definition needlessly verbose so it feels convenient to just use a map. The price of using schemaless includes runtime performance as well as the compiler being unable to warn you about easy (for a compiler) to detect and report data mismatch errors. I have spent a lot of time using lua the way I think many people use clojure. It is quite seductive and does feel very productive at first, but eventually I find that for a sufficiently complex program, I want the compiler to validate my data usage for me.

nucleardog 2y ago

> This is a lot like schemaless databases. The flexibility of not having to fully specify the full schema does not mean that you don't benefit from specifying parts of it? Indeed, if you are indexing things, you have to specify those parts. But it is hard to argue with a straight face that schemaless tools don't have some strong benefits. Let me put on my straightest face. Schemaless tools have no benefits.* As soon as you start _doing_ anything with your data, it has a schema. You may not have declared it to your data store, to a specific piece of code, etc, but your application logic almost definitely makes assumptions about the shape of the data--what fields are and aren't present, the types of values in those fields, and what those values mean. Every piece of your system must either accept undefined behaviour in the face of data which doesn't fit this implicit schema (there are very few non-trivial problems where you can define a meaningful, valid output for all possible inputs), or essentially treat the schemaless parts as hostile and assert its assumptions before every action. The only thing you've done by going "schemaless" is distributing your schema across your systems and pushed the schema validation out to each thing interacting with the schemaless components instead of centralizing it. * Yes, literally storing raw, unstructured data. But as soon as you need to _do_ anything with the data besides regurgitate it you're back to it having some sort of schema.

taeric 2y ago
So you have mainly straw-manned on the word "schemaless" to make an argument. This is akin to attacking dynamic typing by pointing out that most things can be type inferred by the code. That is, you are not wrong in that most data stored is structured in some way; but you are ignoring a ton of the benefit of the schemaless world. As I pointed out, there are benefits to specifying schema. In parts, you /have/ to do so. Specifically if you want your DB to do any work on the data. Indexing/filtering and the like. Now, you do hit largely on the benefits of the "schemaless" world. And that is that it is not undefined how it will store whatever schema you throw at it. You will have restrictions on keys, but otherwise it will store whatever you throw at it. At large, this means you can skip out on a lot of the formality of specifying all schema shapes during development and can take a much more flexible approach in your code than you are forced to if the data layer can't work with you.