3 ms·
Absolutely agree that data cleaning should be at the top -- how someone prioritizes data cleaning is for me, the main litmus test to how effective they are at r
by danso 10y ago
Absolutely agree that data cleaning should be at the top -- how someone prioritizes data cleaning is for me, the main litmus test to how effective they are at real-world data problems. I also agree with how the author summarizes the issue, but he also runs into the same issue I have: data cleaning is such a broad term that it obscures how difficult and important of a problem it is.
For example, some people think data cleaning is "Convert 12-FEB-2012 to 2016-02-12" type problems, and can't believe that such a task would be 80 to 90% of the difficulty in data work (compared to say, learning enough ggplot2 to make a nice chart).
On the other side of the equation, you have people who want to do a JOIN-GROUP-BY aggregate so they can calculate how much "evil" Wall Street money goes to each political candidate, a la OpenSecrets's calculation [0], only to find that the FEC does not classify campaign contributions by industry type or company, nor is the "employer" field filled with normalized entries such as "Evil Wall Street Company" that would lend itself to easy GROUP BY calls. For fucks sake, I've found that executive-level/professor folks can't even spell "Goldman Sachs" and "Berkeley" correctly (even on a typed form)
And that doesn't even scratch the surface of how little this person knows about the data question the purport to answer, or about how the FEC, the American political system, and real life works. Among the data cleaning problems they will have to mitigate are also the 2 hardest problems in computer science (how things are named/classified, and how up-to-date the data is).
I don't have any better ideas at the moment for how to break apart the category of "data cleaning" that reveals the many facets of the problem but also still preserves the interelatedness of the facets. But it's possible to be very good at some of the parts of data cleaning without knowing the rest.
[0] https://www.opensecrets.org/industries/indus.php?Ind=F https://www.opensecrets.org/industries/indus.php?Ind=F
[1] http://www.fec.gov/finance/disclosure/metadata/DataDictionaryContributionsbyIndividuals.shtml http://www.fec.gov/finance/disclosure/metadata/DataDictionar...
- throw_away_777 10y agoI've always disliked the term data cleaning for the reasons you mention - it doesn't tell me anything about what is meant by "cleaning".
- mcrad 10y agoBesides, cleaning implies some entry level gig, something for a QA hack not someone experienced in complex systems. Marketing types love this kinda twist as a way to maintain control (and ensuring failure) of the project. It's like a scapegoat. Just got out of meeting where marketing claims that data "hygiene" is gonna be a priority in 2017.
- zgramana 10y agoI have spent a lot of time talking with customers/prospects about this topic, but I use "data remediation" which I feel brings more accurate and precise connotations, to wit: * Implies that the data is deficient/falls short of expectations. * Implies that the shortcoming currently makes it ineligible to graduate to the next level. * Implies that with hard work and additional time likely it can be made sufficient though still not ideal. * Implies that someone failed to help the data to meet expectations. * Implies that you need special outside expertise, namely someone with the knowledge needed to assess the shortfall, possibly help you clarify your standards, design steps that when followed should result in "good enough" data, and who is able to articulate the remaining weakness(es) which need to be accounted when assessing future suitably of that dataset for a given purpose. * Implies that your data will be stuck in school all summer while their friends are out having so much fun.