3 ms·
It is worse that the articles says -- the data cleaning is often not tracked nor reproducible. First, think about how much data cleaning is not done by a script
by dj-wonk 12y ago
It is worse that the articles says -- the data cleaning is often not tracked nor reproducible. First, think about how much data cleaning is not done by a script or versioned at all. So many people just overwrite the bad data in place. Second, of the data cleaning / ETL that is scripted or partially automated, think about how much is not saved, much less version controlled. As a result, if you ever want to go back and reproduce or share your analysis, good luck!
Doing it better is simple: If you do ETL, version control your code. If you do hand edits, track the before and after.
- joeclark77 12y agoIs anyone actually doing that? I wonder if there's a profession, or if there ought to be, specifically for "data wrangling". On the other hand, maybe you can't separate it from the work of the data scientist... perhaps the data scientist needs to be involved in the data wrangling in order to understand the source better?
- romming 12y ago> perhaps the data scientist needs to be involved in the data wrangling in order to understand the source better? Founder of an ETL startup here. This is exactly what we believe: the end-user of the data should be involved as early in the data pipeline as possible, including the wrangling. If you eliminate the engineer from the ETL process you remove a lot of painful back-and-forth and get more flexible pipelines.
- lmm 12y agoThere are places that do that; at my first job we knew that the ETL code was core business functionality, and version-controlled it appropriately. And we knew that our ability to store raw data reliably was a competitive advantage. But in many cases it's a technical debt issue. In the early startup stages when you're struggling for product-market fit, cleaning or auditing your data is not a good use of time. When a business like that starts to get traction, it finds itself with clients that now want more reliability, scaling issues, and a bunch of data that no-one really remembers where it all came from or how it was generated. A lot of the big-data startups are hitting that stage now.
- stefap2 12y agoI do this with awk or sed scripts. This way I preserve the original data and have an exact and documented steps how the data set was changed. It was ages since I actually went manually into a file to overwrite information.
- scottlocklin 12y agoI can tell you're a practitioner (as opposed to a fly by commenter), as I do the exact same thing!
- itisbiz 12y agoYes and send issues and documentation back to operational teams so they can address root issues in transactional systems. Software selection and system design should have this input by default to avoid implementing data issues.