2 ms·
If the data is a mess, cleaning it up is very much legitimate data science work in my opinion.
by kevinventullo 7y ago
If the data is a mess, cleaning it up is very much legitimate data science work in my opinion.
- dual_basis 7y agoSometimes it can't be cleaned up, or the process of cleaning it up takes too long or is more expensive than the company wants to spend. Sometimes the clean up process is error prone, or leaves you with too little useable data. Sometimes the data really is too noisy and no amount of clean up is really possible. It's definitely true that data clean up is a problem data science can address, but it's not a magic wand.
- blihp 7y agoIt's absolutely part of the process... just not something companies always realize they're signing up for, or the degree to which they are, when they initiate projects. One problem is that the cost of cleaning it up can easily exceed the cost of, and benefit from, the work they planned on doing with the data. Another is that sometimes it's a mess (structurally or the company failed to record key data at point of collection that can't be determined after the fact etc.) to the point of being useless in terms of being able to glean meaningful insight from it. Several of the Data Warehousing projects I've dealt with could be better described as Data Landfills. One can't just dump data into a hole for years, let it rot, and expect goodness when you go back to look at it.