I think that for a lot of A/B testing software (if not all) the statistics is not done correctly. Here is what people do: They keep running the test till they

I think that for a lot of A/B testing software (if not all) the statistics is not done correctly. Here is what people do: They keep running the test till they feel that results are statistically meaningful. That is that they do not fix the N apriori. But that is wrong. Put another way if you wanted to test if a coin was biased you should not stop the test after 10 tosses if you got 9 heads and say the coin is biased. you should decide before hand how many tosses (N) and then conduct the test. And if results are inconclusive you need to flush the data and start again