It's the causation problem. Big Data is getting better at giving us lots of odd correlations -- people buy poptarts before a hurricane. Uber knows where people

It's the causation problem. Big Data is getting better at giving us lots of odd correlations -- people buy poptarts before a hurricane. Uber knows where people are going to go. This information is extremely useful. But what we're missing here is this central fact: correlation data is only good inside a very limited set of preconditions. Once you have a WalMart or an Uber, it helps them operate better. It does not have the ability to create the next WalMart or Uber. That means that Big Data, as it is now, will always be able to continue to optimize within a limited system, but will not be able to see outside of that system. Big Data will not be able to create the next paradigm-changing thing like Uber, because paradigm-changing things are by definition outside the scope of the data already collected. Recently there was a a study published by some MIT students about startups. It ran a bunch of numbers and gave you advice: pick a small name, operate outside the valley, use older workers, and so on. But as somebody pointed out, you really need to have a great idea, spot-on execution, and market traction. If you have that, the rest of it doesn't matter. More to the point, if you have all of those things the MIT guys came up with, they're not going to give you the other things you need. As it turns out, the things you need for a great startup are still fiercely debated -- hence the MIT study in the first place. Correlation does not equal causation. I see this in big companies all of the time. Our projects are running, on average, 100% late! So somebody looks at the data and finds that most of the time is spent testing. What do we need? Better testers, of course! Simply because you can point to a couple of different things that track together does not mean you understand anything. And in IT, unlike Big Data, we do have to worry about causation, because we're inventing the universe every time we ship. You always optimize complex systems from the bottom-up, never from the top-down. Otherwise you're just fooling yourself in various interesting ways. That's true no matter what the system is.