This is necessary with textbook statistical methods which are explicitly based on the assumption of looking at exactly one time. However, modern tools (Optimize

This is necessary with textbook statistical methods which are explicitly based on the assumption of looking at exactly one time. However, modern tools (Optimizely, VWO, Qubit) do not require this. [1] If you use one of these tools it's completely safe to run tests until the test says win. There are still caveats to account for the real world - e.g., only stop the test after an integer number of weeks - but statistically this is a solved problem. [1] The blog post under discussion was written before Optimizely's StatsEngine and VWO's SmartStats.