4 ms·
People are conflating A/B tests are like testing a new revolutionary drug or big discovery. But it isn't Assuming 'B' is the new option, there are 3 possibilit
by raverbashing 8y ago
People are conflating A/B tests are like testing a new revolutionary drug or big discovery. But it isn't
Assuming 'B' is the new option, there are 3 possibilities, A is better than B, A is equivalent to B, A is worse than B
If your p-hacked experiment tells you to change from A to B while the null hypothesis was correct, you didn't get much worse off than you were in the first place. And if your long term metrics were in place then you can get a better measure for your experiment.
Not to mention experimental failures by unaccounted variables
- danieltillett 8y agoYes not to mention that for nearly all A/B tests the size of the effect is minimal anyway. I have found that if the effect is large you don't need statistics and if it is small it doesn't matter.
- dahdum 8y agoA large percentage of experiments I've run were intended to only test if variant was worse than control, we didn't care much how much better the variant may be. Usually these would be positive consumer facing features we were concerned may negatively affect conversion. The switch to Bayesian made that a lot easier to run.