6 ms·
Effective Web Experimentation as a Homo Narrans
- noelwelsh 14y agoGreat post. One of the advantages of the bandit algorithms we favour is that you have to be explicit about your quality metric up-front. This can also be a disadvantage as it can be difficult to tradeoff different metrics, but on balance we find it helpful for our clients. I think one of the main issues with A/B testing, which was also present in the article, is the over-reliance on p-values without considering effect size. The changes show were significant, but none of them had a large effect. The largest was a 9% increase ("Registered") but here we're talking about going from 0.46% to 0.42%, which is pretty much nothing to nothing.
- mcfunley 14y agoYes, that's a great point. Although that test runs on our search pages, so 0.46% of that traffic is about 4,000 registrations. And the difference between 0.46% and 0.42% is about 300 registrations.