3 ms·
Well, if you are running hundreds of tests with 0.05 p-value, you will get plenty of false ok A/A tests, and there isn't much of a reason to expect them to be c
by marcosdumay 4y ago
Well, if you are running hundreds of tests with 0.05 p-value, you will get plenty of false ok A/A tests, and there isn't much of a reason to expect them to be correlated to actual signal on your A/B tests.
A/A tests do test your methodology as you said. But they do not fix a p-value one order of magnitude higher than it should be. (And yeah, I'm aware you know that, but your comment places them on the same context, so it got misleading.)