4 ms·
This is a really good point - blind data analysis without any notion of multiple hypothesis testing leads to false positives. This XKCD is a great illustration
by raviparikh 14y ago
This is a really good point - blind data analysis without any notion of multiple hypothesis testing leads to false positives. This XKCD is a great illustration of that: http://xkcd.com/882/ http://xkcd.com/882/
This is something we're thinking about and are definitely conscious of.
- saosebastiao 14y agoNot just false positives in a confirm/deny scenario. It can also lead to overfitting in a modeling scenario. I once created a model that showed very strongly that US/China exchange rates were a driver of revenue. This model was intuitive based on the market dynamics as we understood them, and they helped provide a strong predictive power which helped reduce operational costs. Over time, this model began to be taken as irrefutable truth among quite a few people in the org. Then when the economy collapsed, we very quickly learned that it wasn't US/China exchange rates that was the strong predictor, but rather something else which correlated with those rates. Not only was the model broken, but it had now breeched trust in the ability of model based forecasts. In other words, if I were a business, I would be bankrupt. I think the best value-add that you can possibly provide with your service is a way of helping your customers understand what is meaningful and what isn't.
- foobarqux 14y agoWhat was the actual underlying correlation?
- saosebastiao 14y agoInflation...specifically using an industrial goods index.
- acgourley 14y agoI encourage you to commission or find web-comic style explanations of common fallacies people could reach by having too much data. Then try and direct them to better strategies. I think that could have a good impact not just on your business but on general perceptions of how to do business analytics.