3 ms·
In 2014 I wrote an article on why Optimizely's approach to AB testing was statistically flawed [1]. I was working at a competitor so I needed to be a bit circum
by martingoodson 6y ago
In 2014 I wrote an article on why Optimizely's approach to AB testing was statistically flawed [1]. I was working at a competitor so I needed to be a bit circumspect.
It was discussed here: https://news.ycombinator.com/item?id=7287665 https://news.ycombinator.com/item?id=7287665
I always wondered how they got away with it for so long.
[1] http://www.datascienceassn.org/sites/default/files/Most%20Winning%20A-B%20Test%20Results%20are%20Illusory.pdf http://www.datascienceassn.org/sites/default/files/Most%20Wi...
- dr_dshiv 6y agoHave you ever given thought to generalizability in A/B tests? I'm surprised there isn't more of a developing science of constructs that generally work...
- richardFINEman 6y agoOptimizely switched to a Frequentist statistics model in 2015, which changed how pretty much all testing companies do stats. Your article was valid, for a full year.
- glangdale 6y agoIt's a bit breathtaking how basic the statistics knowledge is to make this critique (not complaining about the critique, mind you). I'm startled that this was coming as news to anyone.