When you take multiple peeks at the data as it begins to accumulate, you're essentially performing multiple testing. Imagine that you wanted to see if A != B,

When you take multiple peeks at the data as it begins to accumulate, you're essentially performing multiple testing. Imagine that you wanted to see if A != B, with a 95% confidence. To oversimplify, this means that you're willing to accept that 1 in 20 times, you'll incorrectly reject the null and you will consider A different from B even though they are truly the same. If you run 20 independent tests at once, each at a 95% confidence, then by chance you'd expect 1 to reject the null even if they're actually all null. Now, if you repeatedly peek at the data as it accumulates for one test, you're doing something similar (not 100% the same because the tests aren't totally independent, but similar). To again oversimplify, you'd expect, by chance, to see "significant" results 1 in 20 of the times that you look at the data, even if there is no significant result. This is why you need to wait until you've collected all of the necessary data first, or implement a procedure to protect you from incorrectly seeing "significance" when it's not there. You might, for example, require a more stringent confidence level earlier on, and less stringent ones later.