3 ms·
Sounds like a variable cost experiment. Each observation cost x$. Like an A/B split on Google ads. Why keep paying for A when you know B is better already.
by coolcase 1y ago
Sounds like a variable cost experiment. Each observation cost x$. Like an A/B split on Google ads. Why keep paying for A when you know B is better already.
- rrr_oh_man 1y agoGoogle Optimize used to tell you to let an experiment run for one-two weeks (?), exactly because early strong results tend to not don't hold up in the long run. -> https://en.wikipedia.org/wiki/Regression_toward_the_mean https://en.wikipedia.org/wiki/Regression_toward_the_mean
- dr_dshiv 1y agoSeasonality effects, too
- nialse 1y agoSmall samples have more variability than large samples and thus more often show spurious large effects.
- coolcase 1y agoSo you end up with a higher threshold for confidence at p<0.05 ot whatever you want p to be under. Comes out in the maths! Toss a coin 10 times comes up heads 10 times. There is a 1 in 2^10 (approx 1000) that happens by chance for an unbiased coin. I'm convinced it is biased. 20 times I am freaking convinced. I don't need another 1000 tosses.