While interesting, I think that this is more of a mathematical proof for something that people doing any sort of testing should remember: Don't stop before the

While interesting, I think that this is more of a mathematical proof for something that people doing any sort of testing should remember: Don't stop before the test is complete, just because you've gotten an answer. I generally leave my A/B tests up well after I've gotten a significance report, mostly because I'm lazy but also because I know that given enough time and enough entries, the significance reports can change. Especially in the multi-variate tests that Evan wrote about, just because you get one result as significant doesn't preclude other possibilities from also being significant.