5 ms·

my code became much easier to maintain once i stopped thinking of it as writing "algorithms" and "processes" and started thinking of it as a series of type conv

my code became much easier to maintain once i stopped thinking of it as writing "algorithms" and "processes" and started thinking of it as a series of type conversions. structuring what lives where became easier, naming things became systematic and consistent, and writing unit tests became simple.

taberiand 2y ago
I agree; in my experience, pretty much everything is ETL. We take data from one thing, change it a bit, and put it somewhere else. Sometimes, as a treat, we take data from two things, put them together, and then put that somewhere else. Frontend, backend, databases, services, reports, whatever - ETL. In that context, the types and transformations between types are the most important feature. (Everything is actually Category Theory)
- necovek 2y ago
  Except that "ETL" is a terrible name for it: one of those generic acronyms, that even when expanded to "Extract, Transform, Load", needs explanation for at least the "E" and "L" parts. To me, looking at it as "functional" approach instead (data in, operate over that data, data out) is cognitively simpler.
- namaria 2y ago
  I remember working on an ETL pipeline in Airflow years ago and thinking: "we're defining and programming an abstract computer". I guess with general computers everything we do is basically defining nested specific computers. That's, I think, the insight behind SmallTalk and the original concept of objects it used: the objects were supposed to represent computers and the message passing was an abstract network layer.
  - whattheheckheck 2y ago
    https://rachit.pl/post/you-have-built-a-compiler/ https://rachit.pl/post/you-have-built-a-compiler/
tomtom1337 2y ago
This resonates strongly! In Python, I often now find myself declaring Pydantic data structures, and then adding classmethods and regular methods on them to facilitate converting between them. It makes for great APIs (dot-chaining from one type to another), well-defined types (parse, don’t validate) and keeps the code associated with the type.
- whilenot-dev 2y ago
  Exactly! class Profile: ... class User: @classmethod def from_profile(cls, profile: Profile) -> 'User': ... def to_profile(self) -> Profile: ... ...are about all the methods I need in my data records. Three simple rules though: 1. Keep isomorphisms to one class only: Don't put two def to_${OTHER_MODEL_NAME} in each class, instead (like you said) create one static mapping (@classmethod) and one instance mapping 2. Add a mapping to the one class that feels more generalized out of the two: A more generalized data model will probably be used a lot more throughout the application 3. The creation of instances should be pure: If a mapping has side effects and needs to await something then it isn't just a mapping - first resolve all necessary dependencies, then do the mapping

taberiand 2y ago

I agree; in my experience, pretty much everything is ETL. We take data from one thing, change it a bit, and put it somewhere else. Sometimes, as a treat, we take data from two things, put them together, and then put that somewhere else. Frontend, backend, databases, services, reports, whatever - ETL. In that context, the types and transformations between types are the most important feature. (Everything is actually Category Theory)

necovek 2y ago
Except that "ETL" is a terrible name for it: one of those generic acronyms, that even when expanded to "Extract, Transform, Load", needs explanation for at least the "E" and "L" parts. To me, looking at it as "functional" approach instead (data in, operate over that data, data out) is cognitively simpler.
namaria 2y ago
I remember working on an ETL pipeline in Airflow years ago and thinking: "we're defining and programming an abstract computer". I guess with general computers everything we do is basically defining nested specific computers. That's, I think, the insight behind SmallTalk and the original concept of objects it used: the objects were supposed to represent computers and the message passing was an abstract network layer.
- whattheheckheck 2y ago
  https://rachit.pl/post/you-have-built-a-compiler/ https://rachit.pl/post/you-have-built-a-compiler/

tomtom1337 2y ago

This resonates strongly! In Python, I often now find myself declaring Pydantic data structures, and then adding classmethods and regular methods on them to facilitate converting between them. It makes for great APIs (dot-chaining from one type to another), well-defined types (parse, don’t validate) and keeps the code associated with the type.

whilenot-dev 2y ago
Exactly! class Profile: ... class User: @classmethod def from_profile(cls, profile: Profile) -> 'User': ... def to_profile(self) -> Profile: ... ...are about all the methods I need in my data records. Three simple rules though: 1. Keep isomorphisms to one class only: Don't put two def to_${OTHER_MODEL_NAME} in each class, instead (like you said) create one static mapping (@classmethod) and one instance mapping 2. Add a mapping to the one class that feels more generalized out of the two: A more generalized data model will probably be used a lot more throughout the application 3. The creation of instances should be pure: If a mapping has side effects and needs to await something then it isn't just a mapping - first resolve all necessary dependencies, then do the mapping