3 ms·

Thanks for the pointer to CUPED! I wasn't aware of it. After a quick scan, it seems to me that CUPED is supposed to be run at a per-unit level (ie. per user no

by Maro 5y ago

Thanks for the pointer to CUPED! I wasn't aware of it. After a quick scan, it seems to me that CUPED is supposed to be run at a per-unit level (ie. per user normalization), to reduce variance, which seems to be a bit different than the fallacy I describe here (computing lifts from the T and C group's "before" and "after" overall mean separately and subtracting the lifts, which is what I observed and triggered me to write this post). (I wrote the post.)

johnmyleswhite 5y ago
If you work through the math for CUPED, you'll see that only the aggregate correlation term can be interpreted as being "per-unit" -- all of the other terms in the equation are the same terms you use in your post. Insofar as I think your intuition is leading you somewhere, I think it's leading you towards a realization that a "diff in diff" approach rather than regression adjustment can increase variance in some settings. But regression adjustment is provably better in essentially all circumstances: the only settings in which it is ever worse than no adjustment are outlined clearly in https://projecteuclid.org/journals/annals-of-applied-statistics/volume-7/issue-1/Agnostic-notes-on-regression-adjustments-to-experimental-data--Reexamining/10.1214/12-AOAS583.full https://projecteuclid.org/journals/annals-of-applied-statist...
- Maro 5y ago
  Scanning various CUPED related pages, I read that it's a way to reduce the variance, and hence p-value. But CUPED is not changing the lift value (difference [or ratio] in means) between T and C itself (or at least, not in the examples I see). The fallacy I describe computes different means from historic lifts and substracts those. Ie. on the third table, the lift is 4.7%, not -1.6%.