4 ms·
Well, that is what A/B testing is. With a decision rule to try to make that determination in some sensible. Usually the decision rule is stated in terms of a
by btilly 10y ago
Well, that is what A/B testing is. With a decision rule to try to make that determination in some sensible.
Usually the decision rule is stated in terms of a statistical confidence test. Usually the confidence test is done poorly enough that it doesn't mean what people think it does.
And the stopping procedure isn't actually very arbitrary. You choose N based on the most data that you're willing to collect for this experiment. And stop when you're confident about which version will be ahead at that point.
So this procedure leaves you confident of having the best answer that you will get from the most extensive test that your organization is willing to commit to running. And the cost to the organization of running it is capped at sqrt(N) lost conversions.