5 ms·
If you are trying to answer the same question in the same way there isn't much if a difference. Frequentist and bayesian statistics are different paradigms, no
by paulddraper 10y ago
If you are trying to answer the same question in the same way there isn't much if a difference.
Frequentist and bayesian statistics are different paradigms, not different number crunchers.
---
I talk about this with graphs and such in my blog post (https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of-ab-tests-peeking/ https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of...) but here's the short version:
Common question in our line of work: when should I end my A/B test? End too early and I have wrong conclusions, end too long and it's expensive. Classic exploration vs. exploitation.
Frequentist statistics will tell us to suck it up and just choose a sample size, or write down every time we look at the numbers to adjust for peeking effect, or assume the worst case that we always peek and wait a very long time.
Bayesian statistics tells us not to use a single termination condition, but adjust the samplings according to our continuously updating priors. (This is what Google Analytics does. https://support.google.com/analytics/answer/2844870 https://support.google.com/analytics/answer/2844870)
In fact, this turns out to be the mathematically proveable optimal solution for maximizing payoff.
This latter approach is beyond the reach of frequentist methods because it's just a completely different paradigm.
The question of frequentist and bayesian not "which is more correct", but rather "which paradigm best matches the question I have?" If your testing paradigm is fixed sample sizes, choose frequentist. If it is iterative (like a lot of ML), choose bayesian.
- nickhuh 10y agoActually, in the frequentist paradigm you could choose to run a sequential hypothesis test which will end when you've acquired sufficient data[1]. Or, if you want to get fancy you could use a multi-armed bandit approach which is probably optimal in many situations in perhaps a more robust way than many Bayesian methods[2]. Really both can work well. My advice is, use whichever you know well enough to utilize effectively! [1]: https://en.m.wikipedia.org/wiki/Sequential_analysis https://en.m.wikipedia.org/wiki/Sequential_analysis [2]: https://en.m.wikipedia.org/wiki/Multi-armed_bandit https://en.m.wikipedia.org/wiki/Multi-armed_bandit
- kgwgk 10y ago> Actually, in the frequentist paradigm you could choose to run a sequential hypothesis test which will end when you've acquired sufficient data And, as he said, you have to make adjustments to account for these interim analysis.
- paulddraper 10y agoRight, as I said, it can be done with frequentist statistics. This is what Optimizely does (http://pages.optimizely.com/rs/optimizely/images/stats_engine_technical_paper.pdf http://pages.optimizely.com/rs/optimizely/images/stats_engin...). But (1) it is not simple and (2) it is not optimal. --- Agreed 100% about multi-armed bandit which is what I was referring to. And the canonical solutions are in fact Bayesian :) See the Google Analytics link or lookup "Thompson sampling" From your Wikipedia link: "Probability matching strategies are also known as Thompson sampling or Bayesian Bandits, and surprisingly easy to implement if you can sample from the posterior for the mean value of each alternative."
- nickhuh 10y agoOh yeah there are a few Bayesian methods which work great, Bayes-UCB is another. Personally though I think KL-UCB or just plain old UCB would be the ones I'd choose. Like I said earlier, I think these techniques are like programming languages: choose the one you know well enough to get the job done with it.