3 ms·
The data generating process is the random assignment of people to experiment groups. The potential outcomes are fixed: if a person is assigned to one group the
by rwilson4 5y ago
The data generating process is the random assignment of people to experiment groups.
The potential outcomes are fixed: if a person is assigned to one group the outcome is x1; if another, x2. No assumption is made about these potential outcomes. They are not considered random, unless the Population Average Treatment Effect is being estimated. And even in that case, no distribution is assumed. It certainly is not Gaussian for example.
Under random assignment, the observed treatment effect is unbiased for the Sample Average Treatment Effect. So again, the data generating process of interest to the analyst is random assignment.
- kqr 5y agoAssuming you're able to actually achieve truly random participation in the various arms you're trialing, you're right. And it's my fault for not thinking of that as a possibility. Colour me jaded after experiencing very many bad attempts at randomization that actually suffer from Simpson's paradox in various ways!
- rwilson4 5y agoYou're absolutely correct, proper A/B testing has many engineering challenges!