4 ms·
Even in web testing there isn't an equality between false positives and false negatives. Suppose you're Facebook and you decide to test a new landing page on 1
by tedsanders 8y ago
Even in web testing there isn't an equality between false positives and false negatives.
Suppose you're Facebook and you decide to test a new landing page on 1 million users. You roll out the test and notice that after 10,000 users, the new page is killing engagement. Whoops, turns out it had a bug and isn't even loading. Even though no medical patients are dying, this is still a very negative outcome for Facebook. Obviously they shouldn't test on 990,000 more users before fixing the bug, but that's what slavish adherence to pre-registered trial lengths would tell you to do, because it's 'cheating' to notice that there's a problem after the first 10,000 tests.
Some of you might say, sure, for extenuating circumstances like a bug you can break procedure. But in that case I think the logic slides down a slippery slope. What if in the example above instead of a bug we just had a feature performing terribly? In either case, the right move is to end the test early, since you don't need all the data points to measure a strong signal.