3 ms·
> One of the most frustrating results I found is that A/B split tests often resolved into a winner within the sample size range we set; however if I left the sp
by bdjsiqoocwk 2y ago
> One of the most frustrating results I found is that A/B split tests often resolved into a winner within the sample size range we set; however if I left the split running over a longer period of time (eg a year) the difference would wash out.
Doesn't that just mean there's no difference? Why is that frustrating?
Does the frustration come from the expectation that any little variable might make a difference? Should I use red buttons or blue buttons? Maybe if the product is shit, the color of the buttons doesn't matter.
- admax88qqq 2y ago> Maybe if the product is shit, the color of the buttons doesn't matter. This should really be on a poster in many offices.
- cwillu 2y ago“It looks awful, and it works” (apologies to Buckley's)
- sunir 2y agoMy frustration is the a/b split tests never seemed to net to anything in the long term even after confidence was reached. It made me question the entire process; but I understand the math so it’s confusing to me.