Based on very little more than other journalism and recruiters trying to chat me up, here is my understanding. The major hedge funds build in-house big data pl

Based on very little more than other journalism and recruiters trying to chat me up, here is my understanding. The major hedge funds build in-house big data platforms that allow their quantitative traders come up with an idea (what if the price of IBM is 22% predicted by soybean futures in Tokyo on Tuesdays?) and then essentially replay history and see the result. Say you have the soybean-IBM theory. You feed it into the platform. It applies your proposed strategy across billions or trillions of data points. It probably does a lot more than an SQL query -- possibly checking for interactions with other current strategies in use at the firm, running sensitivity analyses, blasting various parts of it with Monte Carlo simulations and so on. A lot of the same tech is also used to perform forecasts of the coming hour, day, week, month or whathaveyou. It comes back with the simulated return. If it's good, and if you think you can disguise your strategy while trading, then and only then can you code it up and deploy it to the trading platform. In this article it's claimed that the models are reviewed by a company founder as well. By now, going on Google's publications about Millwheel and Dataflow, the hedges will already be advanced enough to provide model testing that can converge to a number quickly enough that you can abort the test if it's not shaping up well, thus saving platform time for other uses. Building your own platforms is a mixed blessing. If you the first to do it you can get an enormous advantage because you are the only company with that capability. Later, as technology continues the march of commoditisation, others gain the capability "for free". Now you are at a slowly increasing disadvantage -- again, because you are the only company with that platform.