4 ms·
I’ve never done A/B testing, but is it normal to have the control group and the treatment group historically be so different? The historical difference in aver
by wodenokoto 5y ago
I’ve never done A/B testing, but is it normal to have the control group and the treatment group historically be so different?
The historical difference in average is much larger than the after treatment. Can those groups even be compared after treatment?
- pkoperek 5y agoI think the point is that the control group and treatment groups are in fact the same group (you were giving them the same "service" before starting the experiment) and any difference in the beta_C and beta_T is just random noise. The key part of the argument is that if N goes to infinity the beta_C should be equalt to beta_T (hence it doesn't make much sense to compare them).