3 ms·

Or you got unlucky. There even exist procedures to rerandomize if pre-test metrics are lopsided. You retain the benefits of randomization and it can help a lit

• robocat 5y ago
  Surely getting “unlucky” is a sign of significant variance, which would mean your A/B test needs to be extra robust to detect signal. Presumably, a high standard deviation of multiple randomised pre-groupings could show that your A/B testing is likely to be problematic.