4 ms·

Non-stationary A/B tests

29 points by reckel 4y ago

jldugger 4y ago
I don't quite understand -- how does seasonality not affect both A and B groups equally when randomly chosen?
- ironSkillet 4y ago
  The point is that if your metrics are not stationary, then the variance in observations will seem higher than they really are if you don't account for time. This means you have to collect a lot more data before being able to draw a statistically strong conclusion. They describe a way of adding a time based stratification to the experiment data that doesn't have the weaknesses of simpler methods, like stratifying using a fixed interval size (like day)
  - jldugger 4y ago
    Interesting -- I mostly come from this as an SRE doing canary analysis, where the A and B is basically old and new software version. All of my data is embedded in timeseries data, and stratified by minute =)
- civilized 4y ago
  The paper's intro mentions some of the issues. The biggest one IMO is a Simpson's Paradox issue that arises when (1) the difference between A and B is nonstationary, (2) the probability of being assigned to treatment or control is random at any given instant, but varies over time, being manipulated by dynamic optimization algorithms attempting to maximize signal / profit / etc. Aside - this paper isn't the first time this issue has been noticed, or even the first time it's been addressed with rigorous statistics. Optimizely did it back in 2017-2018: https://support.optimizely.com/hc/en-us/articles/5326213705101-Stats-Accelerator-Acceleration-Under-Time-Varying-Signals https://support.optimizely.com/hc/en-us/articles/53262137051... It would be nice if these authors mentioned the prior art and explained how their work improves upon it.
- MrJohz 4y ago
  Consider a brick and mortar shop doing an A/B test on which poster brings more people into the shop: one of an ice cream, or one of a roaring fireplace. It doesn't matter how randomly you divide your A and B groups, if you run your test in summer, you'll get different results to running it in winter.
FastMonkey 4y ago
What's old is new again I guess? These types of ideas have been around since the 1970s. Block randomization and incorporation of features to model the response surface are already fairly standard well known methods.

29 points by reckel 4y ago

jldugger 4y ago

I don't quite understand -- how does seasonality not affect both A and B groups equally when randomly chosen?

ironSkillet 4y ago
The point is that if your metrics are not stationary, then the variance in observations will seem higher than they really are if you don't account for time. This means you have to collect a lot more data before being able to draw a statistically strong conclusion. They describe a way of adding a time based stratification to the experiment data that doesn't have the weaknesses of simpler methods, like stratifying using a fixed interval size (like day)
- jldugger 4y ago
  Interesting -- I mostly come from this as an SRE doing canary analysis, where the A and B is basically old and new software version. All of my data is embedded in timeseries data, and stratified by minute =)
civilized 4y ago
The paper's intro mentions some of the issues. The biggest one IMO is a Simpson's Paradox issue that arises when (1) the difference between A and B is nonstationary, (2) the probability of being assigned to treatment or control is random at any given instant, but varies over time, being manipulated by dynamic optimization algorithms attempting to maximize signal / profit / etc. Aside - this paper isn't the first time this issue has been noticed, or even the first time it's been addressed with rigorous statistics. Optimizely did it back in 2017-2018: https://support.optimizely.com/hc/en-us/articles/5326213705101-Stats-Accelerator-Acceleration-Under-Time-Varying-Signals https://support.optimizely.com/hc/en-us/articles/53262137051... It would be nice if these authors mentioned the prior art and explained how their work improves upon it.
MrJohz 4y ago
Consider a brick and mortar shop doing an A/B test on which poster brings more people into the shop: one of an ice cream, or one of a roaring fireplace. It doesn't matter how randomly you divide your A and B groups, if you run your test in summer, you'll get different results to running it in winter.

FastMonkey 4y ago

What's old is new again I guess? These types of ideas have been around since the 1970s. Block randomization and incorporation of features to model the response surface are already fairly standard well known methods.