One can commit to a performance metric threshold prior to A/B testing. For example, a rule that states "New, nonmaintenance features must increase revenue for t

One can commit to a performance metric threshold prior to A/B testing. For example, a rule that states "New, nonmaintenance features must increase revenue for the product by at least 1.0%". An objective rule like that neatly resolves the problem of biasing analysis after the test, and doesn't have to necessarily involve any complicated statistics.