3 ms·
It's not some fancy paradox, just a basic mistake. If your "before" metrics are significantly different between control and treatment then you aren't picking th
by paxys 5y ago
It's not some fancy paradox, just a basic mistake. If your "before" metrics are significantly different between control and treatment then you aren't picking the groups correctly.
- tedsanders 5y agoOr you got unlucky. There even exist procedures to rerandomize if pre-test metrics are lopsided. You retain the benefits of randomization and it can help a little when your signal is the same magnitude as your sampling noise.
- robocat 5y agoSurely getting “unlucky” is a sign of significant variance, which would mean your A/B test needs to be extra robust to detect signal. Presumably, a high standard deviation of multiple randomised pre-groupings could show that your A/B testing is likely to be problematic.