4 ms·

Defining basic scalars and giving them good names is useful. Trying to define models (as Taxi defines them) statically is often a source of hidden technical d

by beders 4y ago

Defining basic scalars and giving them good names is useful. Trying to define models (as Taxi defines them) statically is often a source of hidden technical debt as models change over time and different teams in different parts of an org can't agree on what a model should contain. I like that the creator of Taxi is already aware of the anti-pattern lurking behind common domain modeling but doesn't take those lessons learned far enough. Case in point: the field names in models themselves should be the actual Types (again, in the nomenclature of Taxi) `firstName: FirstName` is redundant. It is equivalent to an alias. Instead a model should just be a open set of Types (I would probably call them attributes). That's what Clojure Spec gets right from the get go and leads to much more flexible and open systems. Given a model Person with something like this: firstName: FirstName middleName: MiddleName lastName: LastName ssn: SSN If I only have a FirstName and LastName and don't know the ssn yet, I can't pass these around without having to define an extra model `PersonButWithoutSSN`. I can try to mark ssn as optional, but aren't all these fields optional? And in what combination? Lastly I would advise against enforcing business logic in models, which is what the conditional blocks are effectively doing. Even if just used for data transformation: You'll run the danger of embedding a full scripting language (and you are pretty far already in that direction) Stick to data coercion on the Type level. Any other transformation should be out of scope or defined separately to maximize interop.

martypitt 4y ago
> Trying to define models (as Taxi defines them) statically is often a source of hidden technical debt as models change over time and different teams in different parts of an org can't agree on what a model should contain. I completely agree. My view (and the view of Taxi) is that Types should be shared by teams, but never models - for exactly the reasons you've outlined. Systems need a way of describing their data as a composite of those scalar types - that's what Models provide. Field names are part of that contract, as they tell you structurally where information is. Types describe semantically what that information contains - you need both to semantically document the contract of data. However, if multiple teams start sharing models, you end up with exactly the `PersonButWithoutSSN` situation you've described, which is a BadThing. Our vision is to let data consumers to semantically describe the data they require, in the format they want it. If that doesn't align with how a producer is emitting it, or requires stitching data from multiple producers, that's OK - tooling can enable that, provided you have rich semantic schemas.
throwamon 4y ago
So... This might be a good opportunity for me to clear up some of the doubts I've had for a long time. I don't really work in "teams", I'm an amateur and all projects I've worked on have been just myself, for myself. So please bear with me. Most of the things I do aren't really that important, but there's this one project where I need everything to work perfectly, or else I need to know about issues as soon as possible. The whole thing depends on input I get from scraping, which basically means the data model can change at any time and there's a myriad errors that can happen. Besides from this input data, I also have to check that my "big-picture" assumptions about the model are correct before the actual program runs. In particular, the sample data already determines what values should be possible for most database tables; these tables don't change during program execution. I have a(n also immutable) config file that determines what can be queried from the database. This means I can prove that my config is/isn't correct (e.g. the program should never try to query for an inexistent value) before the program runs. And it should then be possible to turn these values into types to further tighten up the code and my workflow. But creating types from database values just feels all sorts of wrong. Or is it not? All I really want is something that is DRY while also 100% ensuring the consistency of data before I run the program. I guess my idea is: Everything that can be automatically proved about the program before it runs, should be, and it should be stopped on its tracks before running if it's provably incorrect. I don't see how Clojure does that. From everything I've read about Clojure, it sounds like the perfect fit for this kind of thing, but in the end I always conclude I'm too dumb and undisciplined to rely on myself to use it correctly. I make too many mistakes that boil down to assuming something has a different type than it actually has, or that it has a field it actually doesn't, I forget about checking important conditions, etc. And, to be quite honest, I'm too lazy for my own benefit, so I need the system itself to force me to do it the right way, while Clojure assumes you're responsible enough. So I guess my question is: If I switch to Clojure for a critical project, how can I sleep at night?
nudpiedo 4y ago
> `firstName: FirstName` is redundant. It is equivalent to an alias. FirstName is here its own type inheriting String, NOT being a String and that is important in order to enforce not to mixing two different ontological categories represented by the same implementation (STRING). For example, if we follow C style alias of types, FirstName and LastName could be silently exchanged leading to lots of silent errors. I sort of agree with the rest of what you said however more dynamic systems although flexible tend to bring "undesired entropy" into the system.
015a 4y ago
I'm not familiar with Clojure, but: how would that suggestion handle a model like: `hometown: City extends String, livesIn: City extends String` Is the counter-argument that these are poorly specified types, and it should be: `hometown: Hometown extends City extends String, livesIn: LiveIn extends City extends String`? I feel like at some point a balance needs to be struck. Its valuable to name types specifically; but that value can evaporate if they're too specific. It could approach FooFactoryFactoryGeneratorFactory java type-hell, where the names no longer make sense because they have to be globally unique and maximally describe whatever entity its typing. Maximal rigor isn't productive.

by beders 4y ago

martypitt 4y ago

> Trying to define models (as Taxi defines them) statically is often a source of hidden technical debt as models change over time and different teams in different parts of an org can't agree on what a model should contain. I completely agree. My view (and the view of Taxi) is that Types should be shared by teams, but never models - for exactly the reasons you've outlined. Systems need a way of describing their data as a composite of those scalar types - that's what Models provide. Field names are part of that contract, as they tell you structurally where information is. Types describe semantically what that information contains - you need both to semantically document the contract of data. However, if multiple teams start sharing models, you end up with exactly the `PersonButWithoutSSN` situation you've described, which is a BadThing. Our vision is to let data consumers to semantically describe the data they require, in the format they want it. If that doesn't align with how a producer is emitting it, or requires stitching data from multiple producers, that's OK - tooling can enable that, provided you have rich semantic schemas.

throwamon 4y ago

So... This might be a good opportunity for me to clear up some of the doubts I've had for a long time. I don't really work in "teams", I'm an amateur and all projects I've worked on have been just myself, for myself. So please bear with me. Most of the things I do aren't really that important, but there's this one project where I need everything to work perfectly, or else I need to know about issues as soon as possible. The whole thing depends on input I get from scraping, which basically means the data model can change at any time and there's a myriad errors that can happen. Besides from this input data, I also have to check that my "big-picture" assumptions about the model are correct before the actual program runs. In particular, the sample data already determines what values should be possible for most database tables; these tables don't change during program execution. I have a(n also immutable) config file that determines what can be queried from the database. This means I can prove that my config is/isn't correct (e.g. the program should never try to query for an inexistent value) before the program runs. And it should then be possible to turn these values into types to further tighten up the code and my workflow. But creating types from database values just feels all sorts of wrong. Or is it not? All I really want is something that is DRY while also 100% ensuring the consistency of data before I run the program. I guess my idea is: Everything that can be automatically proved about the program before it runs, should be, and it should be stopped on its tracks before running if it's provably incorrect. I don't see how Clojure does that. From everything I've read about Clojure, it sounds like the perfect fit for this kind of thing, but in the end I always conclude I'm too dumb and undisciplined to rely on myself to use it correctly. I make too many mistakes that boil down to assuming something has a different type than it actually has, or that it has a field it actually doesn't, I forget about checking important conditions, etc. And, to be quite honest, I'm too lazy for my own benefit, so I need the system itself to force me to do it the right way, while Clojure assumes you're responsible enough. So I guess my question is: If I switch to Clojure for a critical project, how can I sleep at night?

nudpiedo 4y ago

> `firstName: FirstName` is redundant. It is equivalent to an alias. FirstName is here its own type inheriting String, NOT being a String and that is important in order to enforce not to mixing two different ontological categories represented by the same implementation (STRING). For example, if we follow C style alias of types, FirstName and LastName could be silently exchanged leading to lots of silent errors. I sort of agree with the rest of what you said however more dynamic systems although flexible tend to bring "undesired entropy" into the system.

015a 4y ago

I'm not familiar with Clojure, but: how would that suggestion handle a model like: `hometown: City extends String, livesIn: City extends String` Is the counter-argument that these are poorly specified types, and it should be: `hometown: Hometown extends City extends String, livesIn: LiveIn extends City extends String`? I feel like at some point a balance needs to be struck. Its valuable to name types specifically; but that value can evaporate if they're too specific. It could approach FooFactoryFactoryGeneratorFactory java type-hell, where the names no longer make sense because they have to be globally unique and maximally describe whatever entity its typing. Maximal rigor isn't productive.