4 ms·

I'm interested in how this might work out (or not!) in practice. Turning p=0.05 into p=0.045 is a mortal sin in some circles (see also: p-hacking). I can't se

I'm interested in how this might work out (or not!) in practice. Turning p=0.05 into p=0.045 is a mortal sin in some circles (see also: p-hacking). I can't see how you can look at your data for only "massive signs" without also picking up the smaller signs. Have a computer do it?

croemer 2y ago
The context is A/B testing, so all the data will be on a computer already. It makes sense to abort early when the results are clear. This happens all the time in clinical trials.
- Paul-Craft 2y ago
  Just because it happens all the time doesn't mean it's statistically correct. And, I don't see what the data being on a computer has to do with anything; that also "happens all the time," even in clinical trials.
  - IshKebab 2y ago
    It's statistically correct if your statistics take into account the fact that you might stop early. That's the point. It is more efficient to stop early, but you can't stop early while using statistics that assume you won't.
    - Paul-Craft 2y ago
      As I said, just because it happens all the time does not mean it's statistically correct. I've dug into enough statistical analysis to know that researchers whose field is not statistics simply do not understand statistics enough to know when they're screwing up. Most of them don't even know to apply a correction for applying multiple tests, and you're assuming they'll use statistics that assume they might stop early? Come on. You're imputing a lot more competence to the general case than I believe is actually present, which is my point.
rcxdude 2y ago
The point is that a larger sample size increases precision: the confidence you get rejecting a hypothesis depends on that and the size of the effect. If I tell you I've sampled a process and gotten the values -2, 0, and 15, you can reject the hypothesis that the process is a gaussian process with mean 10000 and standard deviation 0.1 with extreme confidence. Sampling further will almost certainly not change that (in fact you would need an impractical number of samples in this extreme case). The reason you need to tighten your bounds for an early stop is to deal with the fact that you are effectively running multiple experiments: it's possible by chance that you are stopping at a point where the data seems to show an effect by random variation, but because you are not repeating all the samples so far, there's a lot fewer possible outcomes, so the tightening of the bounds that you need is not so much. (p-hacking is simply not applying the appropriate correction for this).
6gvONxR4sf7o 2y ago
I was speaking very roughly. What you do is basically you can peek early just a couple times and stop early if you see p<0.001. You probably won’t, which “leaves you” 0.045 or so to “spend” at the end. Then you have to pretend that when you see p=0.045, that you actually saw p=0.05, and when you see p=0.04999 that it actually said 0.053 or something (non significant at the 0.05 level even though 0.04999<0.05!), so it’s more restrictive at the end… but only by a little! The adjustment at the end is small, so it’s not a big complication and people don’t have to radically change their interpretations like they would with some other alpha spending approaches. And then there’s the big plus that since you only have to change the end interpretation by a little bit, any secondary analyses don’t get crazy complicated. They probably just carry just a little asterisk, rather than coming with the big “this is simple but super untrustworthy” asterisk of secondary analyses after other alpha spending approaches, or having to do something very very expert. If you want to read more, look up alpha spending functions in early stopping. I think the one I’m describing might be called Peto? Sorry i’m being lazy.

croemer 2y ago

The context is A/B testing, so all the data will be on a computer already. It makes sense to abort early when the results are clear. This happens all the time in clinical trials.

Paul-Craft 2y ago
Just because it happens all the time doesn't mean it's statistically correct. And, I don't see what the data being on a computer has to do with anything; that also "happens all the time," even in clinical trials.
- IshKebab 2y ago
  It's statistically correct if your statistics take into account the fact that you might stop early. That's the point. It is more efficient to stop early, but you can't stop early while using statistics that assume you won't.
  - Paul-Craft 2y ago
    As I said, just because it happens all the time does not mean it's statistically correct. I've dug into enough statistical analysis to know that researchers whose field is not statistics simply do not understand statistics enough to know when they're screwing up. Most of them don't even know to apply a correction for applying multiple tests, and you're assuming they'll use statistics that assume they might stop early? Come on. You're imputing a lot more competence to the general case than I believe is actually present, which is my point.

rcxdude 2y ago

The point is that a larger sample size increases precision: the confidence you get rejecting a hypothesis depends on that and the size of the effect. If I tell you I've sampled a process and gotten the values -2, 0, and 15, you can reject the hypothesis that the process is a gaussian process with mean 10000 and standard deviation 0.1 with extreme confidence. Sampling further will almost certainly not change that (in fact you would need an impractical number of samples in this extreme case). The reason you need to tighten your bounds for an early stop is to deal with the fact that you are effectively running multiple experiments: it's possible by chance that you are stopping at a point where the data seems to show an effect by random variation, but because you are not repeating all the samples so far, there's a lot fewer possible outcomes, so the tightening of the bounds that you need is not so much. (p-hacking is simply not applying the appropriate correction for this).

6gvONxR4sf7o 2y ago

I was speaking very roughly. What you do is basically you can peek early just a couple times and stop early if you see p<0.001. You probably won’t, which “leaves you” 0.045 or so to “spend” at the end. Then you have to pretend that when you see p=0.045, that you actually saw p=0.05, and when you see p=0.04999 that it actually said 0.053 or something (non significant at the 0.05 level even though 0.04999<0.05!), so it’s more restrictive at the end… but only by a little! The adjustment at the end is small, so it’s not a big complication and people don’t have to radically change their interpretations like they would with some other alpha spending approaches. And then there’s the big plus that since you only have to change the end interpretation by a little bit, any secondary analyses don’t get crazy complicated. They probably just carry just a little asterisk, rather than coming with the big “this is simple but super untrustworthy” asterisk of secondary analyses after other alpha spending approaches, or having to do something very very expert. If you want to read more, look up alpha spending functions in early stopping. I think the one I’m describing might be called Peto? Sorry i’m being lazy.