2 ms·
The point is that if your metrics are not stationary, then the variance in observations will seem higher than they really are if you don't account for time. Thi
by ironSkillet 4y ago
The point is that if your metrics are not stationary, then the variance in observations will seem higher than they really are if you don't account for time. This means you have to collect a lot more data before being able to draw a statistically strong conclusion. They describe a way of adding a time based stratification to the experiment data that doesn't have the weaknesses of simpler methods, like stratifying using a fixed interval size (like day)
- jldugger 4y agoInteresting -- I mostly come from this as an SRE doing canary analysis, where the A and B is basically old and new software version. All of my data is embedded in timeseries data, and stratified by minute =)