4 ms·

I love the concept of A/A testing here, illustrating that you get apparent results even when you compare something to itself. I can't imagine how A/B tests are

• RyJones 13y ago
  When ExP was a thing at Microsoft, we always ran an A/A test before we did experiments. We'd also do an A/A/B test to make sure the actual experiments were working. http://www.exp-platform.com/Pages/default.aspx http://www.exp-platform.com/Pages/default.aspx
  • darkmighty 13y ago
    There are also some complex problems with assumptions that are infrequently addressed, e.g. maybe if a regular user sees a structural/cosmetic change he is more likely to look at that and click it, while that would fade away in steady-state.
    • RyJones 13y ago
      True. We did everything we could to account for that, though. Someone chooses to clear cookies every time they load a page - not much we could do.
• tel 13y ago
  A/A testing should be used to get accurate estimates for within-sample variance. If you run an A/A/B test then you can calibrate the A/B component to be sensitive w.r.t. the tolerances of real data. And then yeah, I'm sure a lot of successful A/B tests will get washed.
• Homunculiheaded 13y ago
  confidence in your own decisions can also be referred to as a Bayesian prior ;) I've treated the A/B tests I've run pretty much as a case of Bayesian parameter estimation (where the true conversion of A and of B are your parameter). You then get nice beta distributions you can sample from, as well as use the prior to constrain expectations of improvement and also reduce the effects of early flukes in your sampling.
  • mendicantB 13y ago
    Bayesian approaches are probably out of grasp for most small companies. They have a long way to go before being as approachable and easy as frequentist approaches. Schools and the statistics field as a whole need drastic reformation in introductory course offerings that are taken by everyone. Until then, it's A/B, p value <.05, ignore bias and sample size for companies who aren't large enough to have a statistician or data scientist.
    • yummyfajitas 13y ago
      No they aren't. Here is a Bayesian method that is just as easy as any Frequentist one. At my last job, a completely non-technical user who didn't even understand statistical significance used it just fine [1]. http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html The only cost of the Bayesian method is that the bayesian python script is thousands of times slower than the frequentist one. I didn't do benchmarks, but in terms of order of magnitude, the frequentist method might take 1 microsecond while the Bayesian method might take 1 second. [1] He used a less advanced version of the method which used a normal approximation - not that he needed to know the difference.
  • ronaldx 13y ago
    Sorry, but I don't understand how Bayesian statistics could possibly solve the problems described here. Sometimes bad scenarios will get good results, by luck, and sometimes good scenarios will get bad results, by luck. Using more advanced statistical methods doesn't change that these cases are fundamentally indistinguishable.
• ernopp 13y ago
  Full disclosure: I work for Qubit who published this white paper. Just wanted to add that if you have less than a million users you can A/B test for upper funnel goals, effectively measuring if changes improve engagement. Obviously then you have the problem of working out if the engagement translates into more sales but perhaps you're willing to wait longer to find out if a test that improves engagement leads to more revenue in the long run.