Although numerai could control for some of the factors that you mention above, I'm pretty sure they are falling into the data-mining fallacy that most people do

Although numerai could control for some of the factors that you mention above, I'm pretty sure they are falling into the data-mining fallacy that most people do when they approach quantitative trading from the outside world. I completely agree with your skepticism of their approach. While prediction based on data may be valuable in some cases, it isn't robust enough to scale up in any meaningful way. Context matters, like you state above, and most quantitative traders start by taking their contextual knowledge of the markets, and then collecting data on features, and THEN they fit a model to it. Skipping these steps is only going to lead to a bunch of blowups. I doubt they have any meaningful sharpe that they could scale up or publicly defend with the approach they have taken so far. I'd guess they are paying people VC dollars, not actual market profits right now. I do think its cool that they have been able to use homomorphic encryption to solve the problem of wanting anonymize data, but I'm not sure it actually helps in this case.