Exactly. This is why it's important to isolate experiments. Ideally you want what is like a horse stall on your main funnel. Make that stall just 10% of the

Exactly. This is why it's important to isolate experiments. Ideally you want what is like a horse stall on your main funnel. Make that stall just 10% of the total funnel so drastic experiments don't hit too many users. I've worked from Optimizely-style, which is Chi-square / Fisherian, to Bayesian to survival. Each time my req. sample sizes generally came down an order of magnitude (having started at 200,000). That's for conversion rates between 2-6%.