4 ms·
Yes! Came here to write this too. I've been spending the last few years thinking, inquiring and talking about this, and I think there definitely is a field emer
by makmanalp 10y ago
Yes! Came here to write this too. I've been spending the last few years thinking, inquiring and talking about this, and I think there definitely is a field emerging here. There are already some companies trying to think about this stuff, and classes give it lip service but I don't think we've even seen the tip of the iceberg.
It's interesting to talk to different people about data quality and what they think it means, or how they choose to deal with it, and it's all over the place. Some people just mean open and consistent formats, some people have stylistic preferences for data shape, some people talk about accuracy of values, etc etc.
In some ways it's an extension of the thought that the world is inherently noisy, and we've been thinking about that one already, it's just that it turns out you don't need sensor data a la robotics to get noisy data - it's already in the datasets we know and love, and you accumulate more of it, the more sources you pull into your analysis.
- sixdimensional 10y agoData governance and master data management spring to mind as existing ideas along these lines.
- tomrod 10y agoAye, but the locked-down nature of data governance actually fights against using it for insights. I've seen a situation where a small two-way frequency table was requested by Team 1, and it took six _months_ because Team 2 had access to data but no access to metadata, Team 3 had metadata and could help design a query but didn't understand the technical details or run the query, Team 4 had to approve the process, and Team 5 had to review Team 3's query before it could run. In the mean time Team 2 was reorged and Team 1, the original requestors, found upstream sources. Data in a regulatory regime can be excruciatingly difficult, and lend itself to "gut instinct" being used because fear of risk and regulation lock things down too tight to be useful.
- sixdimensional 10y agoI couldn't agree more. Bureaucracy creates silos and stymies sharing and agility. Balance is needed.
- dasboth 10y ago> classes give it lip service I agree with this. It's fine to teach machine learning using the iris dataset, but there is rarely, if ever, a section dedicated to "real" problems. It was a shock to me just how high a percentage of time is spent cleaning data. It is a fundamental skill that is not only underestimated but "undertaught".