6 ms·

In data warehousing and BI, it's MacGyver and data technical debt all the way down. MacGyver because of all the "urgent" reports whipped up for CEO, duplicate c

In data warehousing and BI, it's MacGyver and data technical debt all the way down. MacGyver because of all the "urgent" reports whipped up for CEO, duplicate copies of data and the reports done by consultants who barely understand industry. Data dept because of all the bugs and changes passed down as data from source system.

worldsayshi 9y ago
Does any programming paradigms protect better against data debt? The only way that I can imagine to significantly protect against this would be if there was some way to generate data migrations based on type changes.
- mmsimanga 9y ago
  I don't think there is any programming paradigm that protects against data dept. It is a question of whether the IT team is willing/able to fix the data, with requisite logging of change. This is always the best solution. It often turns into a political nightmare, business insisting that data needs to be corrected. IT not having the resource, time and the change being too risky.
- LtRandolph 9y ago
  I definitely agree that keeping data migration in mind at a foundational level can be very helpful. The ability to run scripts/regexes easily against the data can make it easier to reason about the consequences of your data, too.
- hinkley 9y ago
  I don’t know where it is now, but early versions of Angular encouraged you to isolate all your data debt to the sevice layer. With all the kludges in one place you had a better idea of how bad it was and it was easier to pick a block and insist that it now be handled on the backend.
- paulmd 9y ago
  I don't think there is any technical obstacle or pattern which can prevent dumbasses from shitting things up, humans are just too creative. As soon as you allow any extensibility, someone is going to start shoving integers in as strings, "Y/N" strings as booleans, etc. It would help a lot if there was a well-formed, unambiguous specification for both sides to hold to. Something like the IETF terminology, in terms of MAY/SHALL, specifying things like "true/false" vs "Y/N", etc. Providing sample responses with decent coverage of the possible options is good as well. Then you at least have the leverage to say "aha but the spec says it should be like this, why are you doing it wrong".
  - worldsayshi 9y ago
    > It would help a lot if there was a well-formed, unambiguous specification for both sides to hold to. It does sound like you're describing the schema language part of GraphQL. I think that GraphQL is a great tool for making sure that the right stuff goes in and out. Although it's far from solving all input validation problems. Hmm, perhaps you're describing a different problem. After having worked with GraphQL user input validation at least seems like a manageable problem. There still seems there should be even better methods for handling contagion problems in the data of historical mistakes though.
lmkg 9y ago
It's practically the definition of data warehousing that its whole purpose in life is to deal with everyone else's bullshit. If you want to combine data from different sources, you have to retroactive fix all the mistakes that the data owners made that don't cause issues for them but do cause issues for you.
- bonniemuffin 9y ago
  Story of my life in data science, right there. We work hard to build a culture of shared data ownership, where the data producers have ownership and responsibility of the data they generate, rather than just lobbing garbage over the wall for us to deal with. But it'll always be hard-- as the ultimate the end users of the data, data science/analytics/business ops are always going to care most about its quality.
Unkechaug 9y ago
You're triggering me real hard. 11th hour "I don't care how just make the system get to this number" regardless of the garbage number dumped in. Then you're stuck with a permanent bandaid in the core code that will inevitably screw things up in the future all because of one due date that probably didn't even matter anyway.

worldsayshi 9y ago

Does any programming paradigms protect better against data debt? The only way that I can imagine to significantly protect against this would be if there was some way to generate data migrations based on type changes.

mmsimanga 9y ago
I don't think there is any programming paradigm that protects against data dept. It is a question of whether the IT team is willing/able to fix the data, with requisite logging of change. This is always the best solution. It often turns into a political nightmare, business insisting that data needs to be corrected. IT not having the resource, time and the change being too risky.
LtRandolph 9y ago
I definitely agree that keeping data migration in mind at a foundational level can be very helpful. The ability to run scripts/regexes easily against the data can make it easier to reason about the consequences of your data, too.
hinkley 9y ago
I don’t know where it is now, but early versions of Angular encouraged you to isolate all your data debt to the sevice layer. With all the kludges in one place you had a better idea of how bad it was and it was easier to pick a block and insist that it now be handled on the backend.
paulmd 9y ago
I don't think there is any technical obstacle or pattern which can prevent dumbasses from shitting things up, humans are just too creative. As soon as you allow any extensibility, someone is going to start shoving integers in as strings, "Y/N" strings as booleans, etc. It would help a lot if there was a well-formed, unambiguous specification for both sides to hold to. Something like the IETF terminology, in terms of MAY/SHALL, specifying things like "true/false" vs "Y/N", etc. Providing sample responses with decent coverage of the possible options is good as well. Then you at least have the leverage to say "aha but the spec says it should be like this, why are you doing it wrong".
- worldsayshi 9y ago
  > It would help a lot if there was a well-formed, unambiguous specification for both sides to hold to. It does sound like you're describing the schema language part of GraphQL. I think that GraphQL is a great tool for making sure that the right stuff goes in and out. Although it's far from solving all input validation problems. Hmm, perhaps you're describing a different problem. After having worked with GraphQL user input validation at least seems like a manageable problem. There still seems there should be even better methods for handling contagion problems in the data of historical mistakes though.

lmkg 9y ago

It's practically the definition of data warehousing that its whole purpose in life is to deal with everyone else's bullshit. If you want to combine data from different sources, you have to retroactive fix all the mistakes that the data owners made that don't cause issues for them but do cause issues for you.

bonniemuffin 9y ago
Story of my life in data science, right there. We work hard to build a culture of shared data ownership, where the data producers have ownership and responsibility of the data they generate, rather than just lobbing garbage over the wall for us to deal with. But it'll always be hard-- as the ultimate the end users of the data, data science/analytics/business ops are always going to care most about its quality.

Unkechaug 9y ago

You're triggering me real hard. 11th hour "I don't care how just make the system get to this number" regardless of the garbage number dumped in. Then you're stuck with a permanent bandaid in the core code that will inevitably screw things up in the future all because of one due date that probably didn't even matter anyway.