3 ms·
How can A/B testing tools be improved to prevent p-value hacking? Could it be as simple as declaring your test duration before starting the experiment, and hav
by babl-yc 8y ago
How can A/B testing tools be improved to prevent p-value hacking?
Could it be as simple as declaring your test duration before starting the experiment, and having the tool add an asterisk to your results if you stop the experiment early?
- stenl 8y agoThey should use Bayesian statistics, in which case it doesn’t matter when you stop (more precisely, stopping when you get a result does not bias the outcome; of course running the test longer will make the result more robust). See http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-analysis/ http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-a...
- hackandtrip 8y ago- You must choose the sample size BEFORE - You must choose the significance level BEFORE (0.05 is not always the best choice)
- regularfry 8y agoOptimizely claim to have since fixed this. The paper where they introduce the method is here: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-why-it-matters-and-what-to-do-about-it http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...