It's a bit of a tangent from your main point, but: Bayesian methods try to answer the question of "What is the probability of the model" given some observed da

It's a bit of a tangent from your main point, but: Bayesian methods try to answer the question of "What is the probability of the model" given some observed data There are people arguing that P(M|D) isn't really a meaningful quantity, because the space of models is infinite, and we can't reasonably put a prior on it. I prefer talking about P(M1|D)/P(M2|D): "The data favors model 1 three times more than it favors model 2" The philosophical problem with P(M|D) is the same as distinguishing randomness from non-randomness (cf. Popperian falsifiability): there's no test to prove that something is truly random, all we can say is "We've looked for patterns over and over again, but we haven't found one yet".