5 ms·
+1 I wrote a blog post focusing on the #2 problem mentioned: multiple testing. I simulated the typical mitigating approaches, and comparing them against Thomps
by paulddraper 10y ago
+1
I wrote a blog post focusing on the #2 problem mentioned: multiple testing. I simulated the typical mitigating approaches, and comparing them against Thompson sampling (code in GitHub). https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of-ab-tests-peeking/ https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of...
I'm not a Bayesian fanatic, but given how perfectly A/B test optimization fits in the Bayesian approach, it's a shame it's not yet the de facto standard.
I think the primary reasons are (1) it's not as intuitive (especially for the uninitiated), (2) it's harder to implement an automated feedback mechanism, (3) FUD. E.g. https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html https://www.chrisstucchio.com/blog/2015/dont_use_bandits.htm... lists devastating complications with correct multi-armed bandit tests, and then in fine print admits that traditional tests have all the same complications.
- amasad 10y agoThis is a great read. But left me thinking I'm missing in the fundamentals. Can you recommend a book (or other posts) on statistics fundementals for programmers?
- abecedarius 10y agoIf no one actually knowledgeable answers, I'll say I liked what I read of Doing Bayesian Data Analysis -- early partial draft here: https://faculty.washington.edu/jmiyamot/p548/kruschkejk%20bayesian%20data%20analysis.ch01-06.Pdf https://faculty.washington.edu/jmiyamot/p548/kruschkejk%20ba... I didn't persevere because the R language rubs me the wrong way and I'd be comfier with Python or Haskell or something. http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/ http://camdavidsonpilon.github.io/Probabilistic-Programming-... looks promising in that vein, but I haven't read it.
- saycheese 10y agoGiven most people don't get A/B testing, it's a stretch to me to believe that some that does would know about more complex approaches that require more skill. Firmly believe that there's way more to be gained by more people understanding how to use A/B testing than more complex solutions.
- abecedarius 10y agoIs it really less intuitive? I would've said less familiar. Null-hypothesis significance testing is unintuitive: nonexperts seem to explain it wrongly more often than not. (Like "the p-value is the probability the result was random chance".) Probably both approaches are unintuitive to humans unless they're explained really well.
- paulddraper 10y agoYou're right. True understanding of null hypothesis and p-value isn't easier than Bayesian. Heck, just try to figure out if you need a one-sided or two-sided test. But it's easier to try to black box it and say this number cruncher tells me if these two things are really different. Of course, if you don't understand what's going on, you start p-hacking. Anyway, you're I think you're right.
- abecedarius 10y agoI'd guess the majority of p-hacking is not intentionally fraudulent, yeah. The first few intros to stats I tried to read had trouble sinking in -- it felt opaque and authoritarian somehow. Easy to treat it as a cookbook until I learned of the bayesian approach.
- ted_dunning 10y agoActually, telling you whether things are different isn't the goal. The goal is to make practical business decisions with reasonably high likelihood of being right and, if wrong, having only limited impact. The output of a well done Bayesian analysis can be very easy for business people to understand. Statements like (you have a 60% chance of making just the right decision and a 35% chance of making a decision that is within 3% of right) are easy for most business stake-holders to understand because that is close to how they frame their own decision making process. Laboring to reject a null hypothesis is an unnatural act for most people.