As West is a highly trained physicist, I'll believe he's a bit more statistically careful than that. In the most crude, there's a thing called the Bonferroni co

As West is a highly trained physicist, I'll believe he's a bit more statistically careful than that. In the most crude, there's a thing called the Bonferroni correction which helps to counteract cherry picking effects. Beyond that there are a great deal of other methods to account for overconfidence. If we were not able to identify a small number of relevant patterns in a large sea of options, how would we learn anything? If you've got a mountain of data and are willing to run a lot of models you can indeed identify ones with good tradeoff between prediction and generalization errors. It's very possible that some of your models are weaker than you expect due to cherry picking, but it's even more possible you've identified some useful trends. As you said, he seems to be restating very common knowledge about economies of scale. Since he's not saying anything terribly surprising, I don't think there's much need to get up in arms about multiple comparisons. Moreover he's looking for the on-average rates of growth of these economies of scale and identifying outliers which will likely be interesting. I think it's pretty exciting research.