3 ms·
Data cleaning/munging is a surprisingly pervasive problem in my field ('big data' cancer research). I spend a good fraction of my time hunting down bugs caused
by rgejman 12y ago
Data cleaning/munging is a surprisingly pervasive problem in my field ('big data' cancer research). I spend a good fraction of my time hunting down bugs caused by dirty sources upstream of my scripts. One way to fix this is to become more rigorous about using tools and libraries that check and enforce data correctness and consistency... but it's hard to do this on an individual level. Even if I'm careful about ensuring the correctness of my data, other researchers who send me data may not be as scrupulous. As with any challenge that requires teamwork and constant vigilance (e.g. unit testing), perhaps the data science community needs to invest in tools and computational frameworks that constantly monitor data for correctness and consistency using DDLs and the like.
- mrdmnd 12y agoMy team at Airbnb is working on a tool called "Salus" - named after the Roman goddess of hygiene - that allows you to check the correctness of your data in a number of common, extensible ways. Perhaps this is something that the open-source community at large might be interested in?
- rgejman 12y agoInteresting. I'd love to hear the details.
- Chronic29 12y agoNo you wouldn't.
- christopheraden 12y agoI'd be interested. Any tool that checks correctness is one less thing I have to write before I begin my analyses.
- hmason 12y agoyes.
- palewire 12y agoYes. This is something that could be useful to data journalists, who often have to quickly grip and rip new datasets.
- jcdreads 12y agoI have rolled an extensive (purpose-built) general data quality system at work, and would love for there to be a general purpose tool that does the same sort of thing. I would love to use, contribute to, or otherwise make real an open source version of such a tool. There is an enormous need in several industries for this sort of thing; but most people don't really know they need it yet.
- twone2 12y agoAbsolutely - please keep me updated on progress!
- ehurrell 12y agoDefinitely something I'd be interested in taking a look at, I imagine others would as well!
- joncooper 12y agoYes.
- pbnjay 12y agoSounds intriguing... Is it something rule-based where I have to define things ahead of time, or can it infer stuff - for example highlighting outliers in an automatically detected numeric field?
- seancron 12y agoAbsolutely. Would definitely be interested in hearing more about it.
- kiyoto 12y agoIf cancer research is anything like high frequency trading and time series analysis (which I used to work on), then one heinous thing about corrupted data is inter-dependencies among different data processing tasks: if data coming in "early" in the data workflow topology is garbage, its errors are propagated throughout the dataflow, resulting in multiple days of data detective work, laborious undoing of everything that went wrong, etc. Also, I observe that the data science community does not talk about these issues nearly as much as they should either because 1) it doesn't make a very inspirational topic 2) they are shielded from this kind of issue by a data engineering team.
- rgejman 12y agoAgreed. We need to talk about these issues as a community and come up with effective strategies to prevent data contamination. We've done this for web development and the result was a panoply of tools to validate form input. Now we need to do this for our back-end tools. For instance, statistical programming languages like R, which generally operate on nice tabular data, should come with built in methods to enforce data validity. I should be able to tell R that I expect certain columns to only contain values from 0-1 and no NAs. This is an easy example because languages like R are somewhat dictatorial about how they want you input and process data, but one can imagine the same sort of methods built into the base libraries of general purpose languages. On the other hand, maybe we need to be more rigorous on the whole about data validation. We accept that automated/continuous testing is an effective mechanism for preventing bugs. We need the same automated systems to check data files and flag them when there are issues.
- hadley 12y agoTake a look at the testdat package which provides some tools in the direction.
- deleted 12y ago[deleted]