Standard null hypothesis testing -- including what they're talking about here -- focuses on P(results | random noise is active) You're talking about P(the th

Standard null hypothesis testing -- including what they're talking about here -- focuses on P(results | random noise is active) You're talking about P(the thing we care about | results) The former is more like a simple sanity check. If random noise could have produced what you see, you shouldn't take the results too seriously. The former tells you a little bit about the latter -- which is good, because the latter is what we actually care about. But you can't explicitly compute the latter without making much stronger assumptions like priors and the like. That's why this last step of reasoning is often performed qualitatively. A good stats book to read about all this is Larry Wasserman's http://www.stat.cmu.edu/~larry/all-of-statistics/ http://www.stat.cmu.edu/~larry/all-of-statistics/