3 ms·

The problem is that people often repeatedly check the significance, despite the fact that this test only guarantees a certain false positive rate if you use it

The problem is that people often repeatedly check the significance, despite the fact that this test only guarantees a certain false positive rate if you use it once. If you're planning to stop as soon as you find a positive result you'll need to modify the tests to ensure that the total chance that any test results in a positive is low enough. In general you'll need to keep increasing the significance level as you do more events (if you only plan to test a finite number of times you can keep the significance fixed, but I think this will lead to more false negatives than an 'adaptive' significance). To illustrate, if you have 6000 events and check for 99% significance after every one, then you'd expect about 60 false positives on average. Of course these false positives aren't distributed uniformly, so it's not like you'll always find 60 false positives, however it's not like you'll only find 0 or 6000 either, meaning that (significantly) more than 1% of the time you'll have at least 1 false positive.

tedsanders 10y ago
Checking and stopping is only a problem if you use the inappropriate formula. It's not generally a problem as far as I'm aware. What do you think?
- apathy 10y ago
  It is a problem. You are "spending" some of your error "budget" whenever you peek prior to an endpoint. This is why if you're going to do interim analyses in clinical trials, you have to include it in the design. People got tired of trialists playing games with statistics and patients dying or ineffective drugs making it to market. Now if you want to run a trial as evidence for approval, you need to specify an endpoint, how it will be tested, when, with what alpha (false positive) threshold, and what's the minimum effect size required for this. If you are doing interim analysis, futility, or non-inferiority, you have to write that into the design, too. People can jerk around with subgroup analyses in publications but the FDA won't accept that sort of horse shit for actual approval. And thank heavens for that.
  - tedsanders 10y ago
    In a world with rational actors and free computation, there shouldn't ever be a penalty for having more information about reality. Therefore, the only reason not to peek is that actors are irrational and/or computation is expensive. Honestly, if the first 100 patients die into a 1,000-patient clinical trial, I have zero qualms about making the judgment to stop early, even if it wasn't written into the design. I'm not going to kill 900 people by religiously following bad statistical principles. I think we should be open-minded and understand that sometimes peeking is ok and sometimes it isn't. When the effect is large, you can end earlier. There's no reason to cling to a formula and procedure that requires a fixed number of samples when other methods exist that lack that drawback.
    - contravariant 10y ago
      There's no problem with aborting a test early, for whatever reason. However that doesn't mean you can still draw conclusions from such a test. If you plan to do a trial with 1,000 patients and you stop midway because you've reached statistical significance you run a big risk of claiming a treatment works when it doesn't. Similarly, every test you do has a small probability to give a false positive, the more test you do the bigger the total chance that you'll be jumping to conclusions. Also, the size of the effect is irrelevant since that should already be accounted for by whatever test you do.
      - tedsanders 10y ago
        Any Bayesian analysis should still be valid, despite stopping conditions. You can still draw conclusions from an aborted test. You just have to use valid formulas, and not formulas that assume incorrect counterfactual scenarios. I think it's dogmatic to say stopping is bad and you can't do analysis. In my mind, you totally can. It just needs to be the appropriate analysis. Anyway, I don't even think false positives are the right way to think about this. The framing should be continuous, not binary. The goal is to maximize success, not maximize the odds of picking the better of A or B.
        10y ago
- contravariant 10y ago
  You can probably come up with a proper formula, but you do actually need to put some effort into it to get it right. Of course there's the generic Bonferroni correction where you just divide the significance threshold by the number of tests, but you'll end up with a lot more false negatives (where you just miss the result entirely), and if you want to keep running the test for arbitrary lengths of time it becomes even more difficult (you'll need to keep lowering the threshold). Then again this does give strong guarantees and might work well enough if you don't check the results often. Basically to get the best results you'll need to balance the false negative rate against the false positive rate and the expected length of the test, which is a rather complicated trade off. But I expect someone will have done at least some of the calculations for simple A/B testing.
paulddraper 10y ago
Yep, this is precisely the multiple testing point covered.

tedsanders 10y ago

Checking and stopping is only a problem if you use the inappropriate formula. It's not generally a problem as far as I'm aware. What do you think?

apathy 10y ago
It is a problem. You are "spending" some of your error "budget" whenever you peek prior to an endpoint. This is why if you're going to do interim analyses in clinical trials, you have to include it in the design. People got tired of trialists playing games with statistics and patients dying or ineffective drugs making it to market. Now if you want to run a trial as evidence for approval, you need to specify an endpoint, how it will be tested, when, with what alpha (false positive) threshold, and what's the minimum effect size required for this. If you are doing interim analysis, futility, or non-inferiority, you have to write that into the design, too. People can jerk around with subgroup analyses in publications but the FDA won't accept that sort of horse shit for actual approval. And thank heavens for that.
- tedsanders 10y ago
  In a world with rational actors and free computation, there shouldn't ever be a penalty for having more information about reality. Therefore, the only reason not to peek is that actors are irrational and/or computation is expensive. Honestly, if the first 100 patients die into a 1,000-patient clinical trial, I have zero qualms about making the judgment to stop early, even if it wasn't written into the design. I'm not going to kill 900 people by religiously following bad statistical principles. I think we should be open-minded and understand that sometimes peeking is ok and sometimes it isn't. When the effect is large, you can end earlier. There's no reason to cling to a formula and procedure that requires a fixed number of samples when other methods exist that lack that drawback.
  - contravariant 10y ago
    There's no problem with aborting a test early, for whatever reason. However that doesn't mean you can still draw conclusions from such a test. If you plan to do a trial with 1,000 patients and you stop midway because you've reached statistical significance you run a big risk of claiming a treatment works when it doesn't. Similarly, every test you do has a small probability to give a false positive, the more test you do the bigger the total chance that you'll be jumping to conclusions. Also, the size of the effect is irrelevant since that should already be accounted for by whatever test you do.
    - tedsanders 10y ago
      Any Bayesian analysis should still be valid, despite stopping conditions. You can still draw conclusions from an aborted test. You just have to use valid formulas, and not formulas that assume incorrect counterfactual scenarios. I think it's dogmatic to say stopping is bad and you can't do analysis. In my mind, you totally can. It just needs to be the appropriate analysis. Anyway, I don't even think false positives are the right way to think about this. The framing should be continuous, not binary. The goal is to maximize success, not maximize the odds of picking the better of A or B.
      - 10y ago
contravariant 10y ago
You can probably come up with a proper formula, but you do actually need to put some effort into it to get it right. Of course there's the generic Bonferroni correction where you just divide the significance threshold by the number of tests, but you'll end up with a lot more false negatives (where you just miss the result entirely), and if you want to keep running the test for arbitrary lengths of time it becomes even more difficult (you'll need to keep lowering the threshold). Then again this does give strong guarantees and might work well enough if you don't check the results often. Basically to get the best results you'll need to balance the false negative rate against the false positive rate and the expected length of the test, which is a rather complicated trade off. But I expect someone will have done at least some of the calculations for simple A/B testing.

paulddraper 10y ago

Yep, this is precisely the multiple testing point covered.