3 ms·
The correct way to do this is to adjust for baseline covariates, something that has been done in medicine and other fields for a long time. For A/B-tests specif
by arsemode 5y ago
The correct way to do this is to adjust for baseline covariates, something that has been done in medicine and other fields for a long time. For A/B-tests specifically, the procedure goes by the name CUPED: https://exp-platform.com/cuped/ https://exp-platform.com/cuped/
- heisenzombie 5y agoThat’s very interesting. I’m sure this is a legitimate and useful technique, but (and this is from a place of ignorance about this specific technique) it strikes me as something one should be extremely careful with lest one end up inadvertently p-hacking. It reminds me of another technique used in some fields: Grouping people into “responder” and “non-responder” groups and then looking only at the “responders”… It’s a dangerous technique because it seems sort of reasonable but almost always results in p-hacked rubbish.
- Maro 5y agoThanks for the pointer to CUPED! I wasn't aware of it.