4 ms·
A/B testing is a form of markov-chain-monte-carlo experimentation that makes certain assumptions about the landscape it is trying to find the maximum likelihood
by BlackFly 8y ago
A/B testing is a form of markov-chain-monte-carlo experimentation that makes certain assumptions about the landscape it is trying to find the maximum likelihood in. Specifically, it makes the assumption of monotonicity of the likelihood or that the lack of monotonicity is of the same scale as the changes they are likely to make. A/B testing cannot break out of a local maximum of the likelihood function if the assumptions are incorrect.
A complete rewrite of the application from media heavy to plain is far smaller than the typical stepsize used in A/B testing and has some temporal effects on top of it.
- philwelch 8y agoSure, but you’re implying something super weird, namely that there’s a global maximum of engagement at the “text and CSS” extreme end of the spectrum, but as soon as you add any multimedia at all, even a single img tag, there’s a discontinuous drop and then the metrics improve the more and more you add even more multimedia.
- BlackFly 8y agoI don't think I am saying that. I am saying that the radius of curvature of a peak is potentially greater than the typical step size used in A/B testing. No discontinuity is needed, the landscape could be completely smooth. I'm not suggesting that the extreme of text + css only is what epitomizes a good minimalist page either. I'm just pointing out that it is absolutely not a given that A/B testing has shown that the global maxima is not near it.