3 ms·
All these comments are true. But let's not make the similar error to say that correlations are bad. Sometimes, knowing what caused something is the necessary
by mwexler 15y ago
All these comments are true. But let's not make the similar error to say that correlations are bad.
Sometimes, knowing what caused something is the necessary answer, and for those, a root cause analysis and proper experimental design for validation are important. But sometimes, in business and in life, just knowing that things hang together can be pretty handy.
Correlations are important clues. The entire "recommendation" world, from Amazon's collaborative filtering to Hunch's "everything you might be interested in" are all predicated on correlations.
No argument, saying correlation implies causation is bad. But it's just as bad to say "therefore, correlation is bad". DanielBMarkham's article and this BW.com post both show that it comes down to interpretation of what the data says. It's understanding the limitations of what a number, or a trend, or even a distribution can reveal. It's understanding what regression to the mean actually means, or why we consider a distribution "normal"... and that outliers actually can be profitable.
And it's a recognition that with the democratization of big data, it will get worse before it gets better... but it will get better. 40 years ago, no one ever saw the stock market on the news, or had access to it's ups and downs every second. We now all have a better understanding of stocks (well, ok, that's a bit of a stretch, but you get my drift), and their dangers. Similarly, as we get used to seeing lots more data, and discovering that if you interpret it wrong, bad things happen... well, I expect more folks to ask that next level of questions. Not all, and not much past that... but it will be a start.