3 ms·
> He told me experiments continued until they observed a statistically significant difference between the two conditions. Apparently, if you do the observing t
by Someone 2y ago
> He told me experiments continued until they observed a statistically significant difference between the two conditions.
Apparently, if you do the observing the right way, that is a sound way to do that. https://en.wikipedia.org/wiki/E-values https://en.wikipedia.org/wiki/E-values:
“We say that testing based on e-values remains safe (Type-I valid) under optional continuation.”
- gatopingado 2y agoThis is correct. There's been a lot of interest in e-values and non-parametric confidence sequences in recent literature. It's usually refered to as anytime-valid inference [1]. Evan Miller explored a similar idea in [2]. For some practical examples, see my Python library [3] implementing multinomial and time inhomogeneous Bernoulli / Poisson process tests based in [4]. See [5] for linear models / t-tests. [1] https://arxiv.org/abs/2210.0194 https://arxiv.org/abs/2210.0194 [2] https://www.evanmiller.org/sequential-ab-testing.html https://www.evanmiller.org/sequential-ab-testing.html [3] https://github.com/assuncaolfi/savvi/ https://github.com/assuncaolfi/savvi/ [4] https://openreview.net/forum?id=a4zg0jiuVi https://openreview.net/forum?id=a4zg0jiuVi [5] https://arxiv.org/abs/2210.08589 https://arxiv.org/abs/2210.08589
- ryan-duve 2y agoDid you link the thing that you intended to for [1]? I can't find anything about "anytime-valid inference" there.
- gatopingado 2y agoThanks for noting! This is the right link for [1]: https://arxiv.org/abs/2210.01948 https://arxiv.org/abs/2210.01948