4 ms·
> If this issue of solving data cleaning at scale was easy, I feel that someone would have come up with a very effective answer by now (and as an industry insid
by capkutay 12y ago
> If this issue of solving data cleaning at scale was easy, I feel that someone would have come up with a very effective answer by now (and as an industry inside, I should have heard about it).
Agreed. Even though all these companies in the 'big data' space claim they do everything under the sun because they run on hadoop or spark, data cleansing is still an area people are hesitant to dive into. Rightly so. What surprises me more is that ETL is also widely over-marketed by a lot of new big data companies. I don't think people have paid enough respect to how subtle the 'transform' stage can really be.
- johan_larson 12y agoYes. I work for Informatica on a data integration product -- taking streams of similar but somewhat different data and trying to determine the actual state of things. It's often used for merging customer lists, for example. We have found that determining the best value of truth from incomplete and sometimes conflicting data sources is one of these messy problems with no easy answers. It takes a lot of fiddling with match rules and considerable domain knowledge to get good results. Accordingly, our product is designed to be very flexible and configurable. I suspect improvements will come from making it easier for analysts to see a) the effects of proposed match rule changes and b) what decisions contributed to producing each output record under the current rules.