11 ms·

Any Bayesian analysis should still be valid, despite stopping conditions. You can still draw conclusions from an aborted test. You just have to use valid formul

Any Bayesian analysis should still be valid, despite stopping conditions. You can still draw conclusions from an aborted test. You just have to use valid formulas, and not formulas that assume incorrect counterfactual scenarios. I think it's dogmatic to say stopping is bad and you can't do analysis. In my mind, you totally can. It just needs to be the appropriate analysis. Anyway, I don't even think false positives are the right way to think about this. The framing should be continuous, not binary. The goal is to maximize success, not maximize the odds of picking the better of A or B.

mattkrause 10y ago
Your last sentence is telling. A Bayesian analysis tells you Bayesian things: specifically, this is the most reasonable conclusion one can draw from this data, right now. A frequentist analysis also tells you things, but they are different things. Specifically, frequentists are concerned with...frequencies: if we were to run this procedure many times, here's what we can say about the possible outcomes. There's this persistent meme that all Bayesian methods protect you from multiple comparisons/multiple looks problems. That's not really true. Bayesian methods don't offer any Type I/Type II error control--and why would they, when the notion of these errors doesn't make much sense in a Bayesian framework? You can certainly use Bayesian methods to estimate a parameter's value. However this you cannot repeatedly test your estimates--Bayesian or otherwise--in a NHST-like framework and expect things to work out correctly.