4 ms·

I don't know, that sounds like a complex kind of ingest which could be arbitrarily subtle and diverge over time for legal and bureaucratic reasons. I would kin

I don't know, that sounds like a complex kind of ingest which could be arbitrarily subtle and diverge over time for legal and bureaucratic reasons. I would kind of appreciate having two formats, since what are the odds they would change together? While there may never be a 3rd format, a DRY importer would imply that the source generating the data is also DRY.

TeMPOraL 2y ago
In such case I think I'd go for an internal-DRYing + copy-on-write approach. That is, two identical classes or entry points, one for each format; internally, they'd share all the common code. Over time, if something changes in one format but not the other, that piece of code gets duplicated and then changed, so the other format retains the original code, which it now owns.
- ebolyen 2y ago
  I like that approach.
  - Spivak 2y ago
    I've had the mantra "inheritance is only for code reuse" and it's never steered me wrong.
    - jameshart 2y ago
      Inheritance is only good for code reuse, and it’s a trick you only get to use once for each piece of code, so if you use it you need to be absolutely certain that the taxonomy you’re using it to leverage code across is the right one. All ‘is-a so it gets this code’ models can be trivially modeled as ‘has-a so it gets this code’ patterns, which don’t have that single-use constraint… so the corollary to this rule tends towards ‘never use inheritance’.
      - Spivak 2y ago
        Single use? No way that's what multiple inheritance and mixins are for. Inheritance being only for code reuse is explicitly about not creating a taxonomy. No more is-a just, "I need this code here." Hey this thing behaves like a mapping inherit from the MutipleMapping and get all the usual mapping methods for free. Hey this model needs created/updated_at, inherit from ChangeTracking and get those fields and helper methods for free. Has-a doesn't make sense for code like the literal text reuse. It makes sense for composition and encapsulation. Edit: I'm now realizing that Python has one of the only sane multiple inheritance implementations. It's no wonder the rest of y'all hate it.
    - zeroCalories 2y ago
      It seems like OP is describing a shared interface, not necessarily inheritance.
- jeremyjh 2y ago
  This is really good advice and a great way to think about it.
- chii 2y ago
  I believe this very method is very common in games - you have similar logic for entities, but some have divergences that could occur in unknown ways after playtesting or future development. Tho if done haphazardly by someone inexperienced, you might end up with subtle divergences that might look like they're meant to be copies, and debugging them in the future by another developer (without the history or knowledge) can get hard. Then someone would wonder why there are these two very similar pieces of code, and mistakenly try to DRY it in the hopes of improving it, causing subtle mistakes to get introduced...
  - cies 2y ago
    I prefer the FP approach of separating data and logic. you could end up with a box of functions (logic) that can be reused by the different "entities". Last time i checked the FP world is slowly producing ECS frameworks that are needed to make the game performant. They used to be nearly C++ (or OO) exclusive.
    - trealira 2y ago
      Is an entity component system really functional programming? I had the sense that functional programming was more about writing functions that are pure and referentially transparent, and making data immutable normally, which can make code simpler and more modular. It tends to use higher-order functions (recursive operators) more than direct recursion, because it's easier to verify correctness then. Rather than imperative loops and mutation, the meat of the program consists of the composition of many different functions, both small and large. Entity component systems are pretty cool, as is functional programming, but I don't see the relation. In addition, object-oriented languages seem well-suited to making entity component systems. There are some tutorials on them in different object-oriented programming languages: C++: https://austinmorlan.com/posts/entity_component_system/ https://austinmorlan.com/posts/entity_component_system/ C#: https://matthall.codes/blog/ecs/ https://matthall.codes/blog/ecs/ Common Lisp: https://edoput.it/2023/11/19/data-oriented-clos.html https://edoput.it/2023/11/19/data-oriented-clos.html
- chipdart 2y ago
  > In such case I think I'd go for an internal-DRYing + copy-on-write approach. I agree. The primary risk of presented by DRY is tight coupling code which only bears similarities at a surface level. Starting off by explicitly keeping the externa bits separate sounds like a good way to avoid the worst tradeoff. Nevertheless I still prefer the Write Everything Twice (WET) principle, which means mostly the same thing, but following a clear guideline: postpone all de-duplication efforts until it's either obvious there's shared code (semantics and implementation) in >2 occurrences, and always start by treating separate cases as independent cases.
Stratoscope 2y ago
Good point. This may be a case where domain knowledge is helpful. One of the reasons they brought me in on this project is that besides knowing how to wrangle data, I'm also an experienced pilot. So I had a good intuitive sense of the meaning and purpose of the data. The part of the data that was identical is the description of the airspace boundaries. Pilots will recognize this as the famous "upside down wedding cake". But it's not just simple circles like a wedding cake. There are all kinds of cutouts and special cases. Stuff like "From point A, draw an arc to point B with its center at point C. Then track the centerline of the San Seriffe River using the following list of points. Finally, from point D draw a straight line back to point A." The FAA would be very reluctant to change this, for at least two reasons: 1. Who will provide us the budget to make these changes? 2. Who will take the heat when we break every client of this data?
- ebolyen 2y ago
  I see, so it's a procedural language that is well understood by those who fly (not just some semi-structured data or ontology). This is a great example of the advantage of domain experience. Thanks for sharing!
  - Stratoscope 2y ago
    > a procedural language that is well understood by those who fly That is a great way to describe it! Of course it is all just rows in a CSV file, but yes, it is a set of instructions for how to generate a map. In fact the pilot's maps were being drawn long before the computer era. Apparently the first FAA sectional chart was published in 1930! So the data format was derived from what must have been human-readable descriptions of what to plot on the map using a compass and straightedge. I just remembered a quirk of the Australian airspace data. Sometimes they want you to draw a direct line from point F to point G, but there were two different kinds of straight lines. They may ask for a great circle, a straight path on the surface of the Earth. Or a rhumb line, which looks straight on a Mercator projection but is a curved path on the Earth. You would often have some of each in the very same boundary description! For anyone curious about this stuff, I recommend a visit to your local municipal airport and stop by the pilot shop to buy a sectional chart of your area.
bcrosby95 2y ago
I don't know. I've seen this approach for projects before go bad - people didn't want to DRY because they might diverge. Except they never did. Our 3rd+ scenarios we abstracted. But what basically ended up happening was we had 2 codebases: 1 for that non-DRY version, and then 1 for everything else. The non-DRY version limped along and no one ever wanted to work on it. The ways it did things were never updated. It was rarely improved. It was kinda left to rot.
- kiitos 2y ago
  I have seen far more problems caused by premature or over-use of DRY than the opposite. Undoing an abstraction is always far more difficult than the pains caused by not having one in the first place.
- jononor 2y ago
  Why wasn't the original implementation swapped for the new one? The unwillingness/inability to do that seems to be most likely the core of the issues here?
  - bcrosby95 2y ago
    The majority of our business was through the 1st implementation. Because of that it was the base we used to refactor into a more abstract solution for further scenarios. It was never deemed "worth it" to transition the 2nd non-DRY version. Why refactor an existing implementation if its working well enough and we could expand to new markets instead?
    - jononor 2y ago
      Yes, why do it? :p I mean, there are pros and cons - costs and benefits. And I can see both scenarios where it is better to spend the time on something else (that has better chance of bringing in money), and cases where it would be the right thing to do the cleanup (maybe original is just about to fall apart, or the new has straight up benefits to the business, or the act of doing it will greatly improve testing/QA in a critical area, etc). Writing it DRY in the first place would also have costs, including the alternative costs. Would it have been better to take those there and then?
mannykannot 2y ago
I vaguely recall Fred Brooks, in The Mythical Man-Month, using a somewhat similar situation, but involving the various US states' income tax rules and their relationship to the federal tax, as an example in order to make some sort of point (that point being 'know your data', IIRC.) In a situation where there is a base model with specific modifications - which is, I feel, how airspace regulation mostly works - then I suspect that a DRY approach would make it easier to inspect and test, so long as it stays that way.

TeMPOraL 2y ago

In such case I think I'd go for an internal-DRYing + copy-on-write approach. That is, two identical classes or entry points, one for each format; internally, they'd share all the common code. Over time, if something changes in one format but not the other, that piece of code gets duplicated and then changed, so the other format retains the original code, which it now owns.

ebolyen 2y ago
I like that approach.
- Spivak 2y ago
  I've had the mantra "inheritance is only for code reuse" and it's never steered me wrong.
  - jameshart 2y ago
    Inheritance is only good for code reuse, and it’s a trick you only get to use once for each piece of code, so if you use it you need to be absolutely certain that the taxonomy you’re using it to leverage code across is the right one. All ‘is-a so it gets this code’ models can be trivially modeled as ‘has-a so it gets this code’ patterns, which don’t have that single-use constraint… so the corollary to this rule tends towards ‘never use inheritance’.
    - Spivak 2y ago
      Single use? No way that's what multiple inheritance and mixins are for. Inheritance being only for code reuse is explicitly about not creating a taxonomy. No more is-a just, "I need this code here." Hey this thing behaves like a mapping inherit from the MutipleMapping and get all the usual mapping methods for free. Hey this model needs created/updated_at, inherit from ChangeTracking and get those fields and helper methods for free. Has-a doesn't make sense for code like the literal text reuse. It makes sense for composition and encapsulation. Edit: I'm now realizing that Python has one of the only sane multiple inheritance implementations. It's no wonder the rest of y'all hate it.
  - zeroCalories 2y ago
    It seems like OP is describing a shared interface, not necessarily inheritance.
jeremyjh 2y ago
This is really good advice and a great way to think about it.
chii 2y ago
I believe this very method is very common in games - you have similar logic for entities, but some have divergences that could occur in unknown ways after playtesting or future development. Tho if done haphazardly by someone inexperienced, you might end up with subtle divergences that might look like they're meant to be copies, and debugging them in the future by another developer (without the history or knowledge) can get hard. Then someone would wonder why there are these two very similar pieces of code, and mistakenly try to DRY it in the hopes of improving it, causing subtle mistakes to get introduced...
- cies 2y ago
  I prefer the FP approach of separating data and logic. you could end up with a box of functions (logic) that can be reused by the different "entities". Last time i checked the FP world is slowly producing ECS frameworks that are needed to make the game performant. They used to be nearly C++ (or OO) exclusive.
  - trealira 2y ago
    Is an entity component system really functional programming? I had the sense that functional programming was more about writing functions that are pure and referentially transparent, and making data immutable normally, which can make code simpler and more modular. It tends to use higher-order functions (recursive operators) more than direct recursion, because it's easier to verify correctness then. Rather than imperative loops and mutation, the meat of the program consists of the composition of many different functions, both small and large. Entity component systems are pretty cool, as is functional programming, but I don't see the relation. In addition, object-oriented languages seem well-suited to making entity component systems. There are some tutorials on them in different object-oriented programming languages: C++: https://austinmorlan.com/posts/entity_component_system/ https://austinmorlan.com/posts/entity_component_system/ C#: https://matthall.codes/blog/ecs/ https://matthall.codes/blog/ecs/ Common Lisp: https://edoput.it/2023/11/19/data-oriented-clos.html https://edoput.it/2023/11/19/data-oriented-clos.html
chipdart 2y ago
> In such case I think I'd go for an internal-DRYing + copy-on-write approach. I agree. The primary risk of presented by DRY is tight coupling code which only bears similarities at a surface level. Starting off by explicitly keeping the externa bits separate sounds like a good way to avoid the worst tradeoff. Nevertheless I still prefer the Write Everything Twice (WET) principle, which means mostly the same thing, but following a clear guideline: postpone all de-duplication efforts until it's either obvious there's shared code (semantics and implementation) in >2 occurrences, and always start by treating separate cases as independent cases.

Stratoscope 2y ago

Good point. This may be a case where domain knowledge is helpful. One of the reasons they brought me in on this project is that besides knowing how to wrangle data, I'm also an experienced pilot. So I had a good intuitive sense of the meaning and purpose of the data. The part of the data that was identical is the description of the airspace boundaries. Pilots will recognize this as the famous "upside down wedding cake". But it's not just simple circles like a wedding cake. There are all kinds of cutouts and special cases. Stuff like "From point A, draw an arc to point B with its center at point C. Then track the centerline of the San Seriffe River using the following list of points. Finally, from point D draw a straight line back to point A." The FAA would be very reluctant to change this, for at least two reasons: 1. Who will provide us the budget to make these changes? 2. Who will take the heat when we break every client of this data?

ebolyen 2y ago
I see, so it's a procedural language that is well understood by those who fly (not just some semi-structured data or ontology). This is a great example of the advantage of domain experience. Thanks for sharing!
- Stratoscope 2y ago
  > a procedural language that is well understood by those who fly That is a great way to describe it! Of course it is all just rows in a CSV file, but yes, it is a set of instructions for how to generate a map. In fact the pilot's maps were being drawn long before the computer era. Apparently the first FAA sectional chart was published in 1930! So the data format was derived from what must have been human-readable descriptions of what to plot on the map using a compass and straightedge. I just remembered a quirk of the Australian airspace data. Sometimes they want you to draw a direct line from point F to point G, but there were two different kinds of straight lines. They may ask for a great circle, a straight path on the surface of the Earth. Or a rhumb line, which looks straight on a Mercator projection but is a curved path on the Earth. You would often have some of each in the very same boundary description! For anyone curious about this stuff, I recommend a visit to your local municipal airport and stop by the pilot shop to buy a sectional chart of your area.

bcrosby95 2y ago

I don't know. I've seen this approach for projects before go bad - people didn't want to DRY because they might diverge. Except they never did. Our 3rd+ scenarios we abstracted. But what basically ended up happening was we had 2 codebases: 1 for that non-DRY version, and then 1 for everything else. The non-DRY version limped along and no one ever wanted to work on it. The ways it did things were never updated. It was rarely improved. It was kinda left to rot.

kiitos 2y ago
I have seen far more problems caused by premature or over-use of DRY than the opposite. Undoing an abstraction is always far more difficult than the pains caused by not having one in the first place.
jononor 2y ago
Why wasn't the original implementation swapped for the new one? The unwillingness/inability to do that seems to be most likely the core of the issues here?
- bcrosby95 2y ago
  The majority of our business was through the 1st implementation. Because of that it was the base we used to refactor into a more abstract solution for further scenarios. It was never deemed "worth it" to transition the 2nd non-DRY version. Why refactor an existing implementation if its working well enough and we could expand to new markets instead?
  - jononor 2y ago
    Yes, why do it? :p I mean, there are pros and cons - costs and benefits. And I can see both scenarios where it is better to spend the time on something else (that has better chance of bringing in money), and cases where it would be the right thing to do the cleanup (maybe original is just about to fall apart, or the new has straight up benefits to the business, or the act of doing it will greatly improve testing/QA in a critical area, etc). Writing it DRY in the first place would also have costs, including the alternative costs. Would it have been better to take those there and then?

mannykannot 2y ago

I vaguely recall Fred Brooks, in The Mythical Man-Month, using a somewhat similar situation, but involving the various US states' income tax rules and their relationship to the federal tax, as an example in order to make some sort of point (that point being 'know your data', IIRC.) In a situation where there is a base model with specific modifications - which is, I feel, how airspace regulation mostly works - then I suspect that a DRY approach would make it easier to inspect and test, so long as it stays that way.