3 ms·
> Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent,
by dkarl 6y ago
> Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name.
That's kind of what I understand as well, but the data science folks pitched it in a slightly more positive way, like, "Please don't limit us just to the data you have time to nicely structure and validate. We want all of it. It doesn't matter if a column is getting truncated to three characters or columns are mislabeled or there are amounts in dollars and euros mixed together; we can still get signal out of it."
- zekrioca 6y agoMore or less how a proof-of-concept for new type of wine turns out to be sold as Champagne..
- loopz 6y agoAs another poster said, if it's the raw data, the raw deal, it's the bets source and nice to have (but could become expensive, so better it is paid for in scale). If it's something else, it's the antithesis of data science.
- freewilly1040 6y ago> we can still get signal out of it Today you can, but then when the app owner drops a column (or worse, stop populating it!) in a month that signal will break, and the data lake maintainer will be in the unenviable position of navigating the completely undocumented dependency.
- dmpetrov 6y agoThat's might be fine. Anyway, you need to retrain your models quite often due to the concept drift. Solving data problems like this is just a part of model retraining (and concept drift).