It's also worth pointing out that you can safely peek and make an early decision if you insist on a higher confidence interval for early stops. The exact numbe

It's also worth pointing out that you can safely peek and make an early decision if you insist on a higher confidence interval for early stops. The exact numbers depend on how often you test the confidence interval, but a multiple of 5-10 is for early measures will usually give reasonable results. E.g., let's say you ultimately want a 98% confidence interval (2% chance of error). A 10x improvement is a .2% change of error, or 99.8%. Depending on how often you are sampling, an early result with 99.8% confidence can be equally accurate as a later results with 98% confidence.