I do not think you can just stop the test when you find something you think is significant http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.

I do not think you can just stop the test when you find something you think is significant http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html 'If instead of deciding ahead of time, “this experiment will collect exactly 1,000 observations,” you say, “we’ll run it until we see a significant difference,” all the reported significance levels become meaningless. ' Reading the stats.py it looks like you might be doing this.