10 ms·
I think a reason that leads to the need of something like a 'data lake' or anything that looks like a messy pile of incoherent data, is the difference between h
by cpard 6y ago
I think a reason that leads to the need of something like a 'data lake' or anything that looks like a messy pile of incoherent data, is the difference between how data scientists and traditionally data analysts deal with noise in the data. Most BI tasks require the data to be as clean as possible, it's important to be aware of the quality of the data before you calculate your MRR for example. On the other side, data scientists deal with noise more as a parameter of the models they are building. This difference leads to different requirements on how the pipelines should operate. In such a context, having a messy pile of data might help increase the velocity and the independence of some teams inside the company.
- jamesblonde 6y agoNoise yes, null or missing values will crash your training and out-of-distribution values (which your schema will not help you with!) will be even worse, hurting performance but impossible to find.
- cpard 6y agoTotally agree on that, both have still to work with data quality but in different ways and with a different mindset.