8 ms·
Bayesians moving from defense to offense
- tomrod 3y agoHuh. Wouldn't publication bias dramatically impact a Bayesian's conclusions?
- ta988 3y agoDepends how you model it. I would say not taking into account the publication bias if you have ways to measure or at least estimate its distribution is a bigger mistake to make.
- ChadNauseam 3y agoYes, it should affect anyone's conclusions unless they know exactly how much publication bias there is and then account for it
- pacbard 3y agoYes, possibly. Publication bias is a known problem when using prior work (see [1] for some references on the problem and potential solutions). One of the strengths of Bayesian methods is that you can use a less informative prior to model your skepticism of prior work. [1]: https://www.sciencedirect.com/topics/medicine-and-dentistry/funnel-plot#:~:text=In%20a%20funnel%20plot%2C%20the,plots%20may%20suggest%20publication%20bias https://www.sciencedirect.com/topics/medicine-and-dentistry/....
- epgui 3y agoYes, but no more than a non-Bayesian’s. The difference is mainly that the Bayesian approach makes more of its assumptions explicit.
- jvans 3y agoThis. Bayesian assumptions are explicit and you'll be forced to defend your priors. Frequentist assumptions like homoscedasticity are implicit and rarely challenged. I wouldn't be surprised if many people using frequentist methods have no idea what assumptions need to be true for the methods to be valid. In a Bayesian framework you can't avoid the assumptions
- ajkjk 3y agoThat's why it's for constructing a prior that you change later; your study itself should update the prior in the direction of being more correct. But it's not like you can do better than "the best available prior" (by definition).
- mjfl 3y agoI've never heard anyone promote frequentist statistics. I've only ever heard Bayesians on the 'offense' talking about why their framework is better. To be clear I am someone who doesn't really understand the faultlines.
- analog31 3y agoIndeed, when I took statistics in college, nearly 40 years ago, we proved Bayes theorem, and worked through several applications, so it could hardly have been regarded as controversial.
- adastra22 3y agoIt’s not. Statistics is statistics. This controversy is new and contrived.
- fbdab103 3y agoTell that to Ronald Fisher, who made it a point to deride Bayesian stats wherever there was an opportunity.
- jvans 3y agoThe controversy is decades old and largely resolved. Both methods work well on large enough data sets, Bayesian methods are far superior on small datasets
- adastra22 3y agoThis statement doesn’t really make sense to me. A frequentist approach assumes a random (or noisy) underlying process and characterizes its probability. A Bayesian approach assumes a fixed but unknown process and updates the probability a given model matches that reality. It is ultimately the same math, if you are comparing apples to apples. Just different ways of looking at a problem. Sometimes one is better suited than the other. It’s like as if physicists were arguing over whether Cartesian or Polar coordinates were better. It’s the same damn physics, just expressed differently. In some problems one approach is easier to work with than the other, and can even make seemingly intractable problems solvable. But that doesn’t mean the other approach was “wrong.”
- hackandthink 3y agoLink to the first quoted paper: http://www.stat.columbia.edu/~gelman/research/published/pval_RCTs_rev3.pdf http://www.stat.columbia.edu/~gelman/research/published/pval...
- pil0u 3y agoWhat resources would you recommend to implement an A/B test evaluation framework using a Bayesian approach? I would love to embrace (or at least try to) such a new approach, but it feels like without a PhD in stats it's hard to get it started.
- jvans 3y agoStatistical rethinking by McElreath is an exceptional resource for Bayesian modeling. It's very accessible and gives you some good examples to work through. Some very basic Bayesian models can go a long way towards making informed decisions for a/b tests
- plants 3y agoSpecifically for A/B or A/B/N testing, you can use a beta-bernoulli bandits, which give you confidence about which experience is best and will converge to an optimal experience faster than your standard hypothesis test. Challenges are that you have to frequently recompute which experience is best and thus, dynamically reallocate your traffic. They also only works on a single metric, so if your overall evaluation criterion isn’t just something like “clickthrough rate”, this type of testing becomes more difficult (if anyone else knows how multiple competing metrics are optimized with bandits, feel free to chime in).
- abhgh 3y agoBeta-Bernoulli multi-armed bandits (BB-MAB) are definitely a good way to get started on a Bayesian version of A/B testing where you've the additional benefit that your population is dynamically allocated to the most performant option (actually this dynamic allocation makes it similar to interim analysis [1] rather than vanilla A/B testing). There are some caveats though - and I mention these from the experience of running such solutions on a large scale in production. First, BB-MAB can't adapt to context by design. They only look at click/no-click behavior across the population. So, if your population has two distinct segments - youth and elderly - who behave very differently wrt purchases, the BB-MAB won't pick a different winning advt. per group; its blind to these groups. The solution is to use something like a contextual MAB - which assimilates user features (or whatever you might throw at it) into the MAB. There are simple ways to adapt simple MABs to the contextual setup [2] (in my experience, these can also be effective) but, of course, the literature in this area is wide and deep. A second caveat is that if the ratio of the size of the pool of advts. to the number of impressions is high, the BB-MAB won't converge or converge to a good optima; the search space is simply too large relative to the data. In cases like this it becomes important to begin with the right Beta priors, instead of the standard recipe of starting with a Beta that looks like a uniform distribution. [1] https://en.wikipedia.org/wiki/Interim_analysis https://en.wikipedia.org/wiki/Interim_analysis [2] https://arxiv.org/abs/1811.04383 https://arxiv.org/abs/1811.04383
- clircle 3y agoI don't really see Frequentism and Bayes to be in conflict. You use them to answer different questions. I wouldn't test a point null hypothesis with a Bayesian model.
- kneel 3y agoIt honestly just seems like an abstract debate to claim that your logical framework is somehow superior. Different tools for different datasets
- mitthrowaway2 3y agoI think the Bayesians argue that you would, so there does seem to be a conflict.
- clircle 3y agoI think most Bayesians would claim that point null hypothesis tests are immoral or something like that, rather than propose to do them with a Bayesian model (thus avoiding measure zero problems)
- zestyping 3y agoI don't know when Bayesians have ever been on defense. They've always been on offense.
- carabiner 3y agoWe will win.
- deleted 3y ago[deleted]
- paulsutter 3y agoBad reasoning trumps arcane methodology debates every time
- vismwasm 3y agoI never really got Bayesian statistics to be honest. - When sample size grows, frequentist and bayesian (if the prior is not too restrictive) point estimates seem to converge to each other anyway - The distribution of your point estimate (frequentist) vs. the estimated distribution (bayesian) also don't seem to differ too much either - When the sample size is small the Bayesian prior dominates - Interestingly, when I see Bayesians simulate random data (to introduce the concepts on this data) they usually assume a true parameter value. E.g. when sampling from Y = a + b * X + e, they'll assume fixed, true values of a and b and not random variables - which is a frequentist assumption! So far I've never seen e.g. b being sampled from Normal(mu=2, sigma=1) instead of just setting b=2. - The frequentist assumption of a true population value which we try to estimate just makes sense to me. For example there is a true mean income over the working population. It's not a random variable but a fixed value which can be computed if we just asked every single working person for their income and then compute the mean over all values. I tried getting into Bayesian stats but honestly it just seems overkill for most cases. For a simple regression computing b_hat = inv(XX')Y is just faster and easier than numerically sampling traces. Bayesian forces you to think about the data generating process - I appreciate that, but you need to the same when it comes to frequentist stats, it's just a little less obvious.
- enriquto 3y ago> When sample size grows, frequentist and bayesian [...] estimates seem to converge to each other anyway Yes. And so? Bayesians would argue (and I quote) that "the interesting limit in statistics is when the number of samples tends to one. The limit when the number of samples tends to infinity is completely useless." > I tried getting into Bayesian stats but honestly it just seems overkill for most cases. There are 3 black balls and 7 white balls in an opaque bag. How likely is it to pick a black ball? Bayesian statistics gives a straightforward answer (you just assume an uninformative prior and perform a computation). But frequentist statistics starts to argue about an infinite number of replicas of your own universe and other nonsensical constructions. Not sure that the Bayesian approach is overkill in that case...
- Dylan16807 3y ago
- jabowery 3y agoIt's always struck me as rather strange that since the motive for creating any kind of model is to calculate predictions, and that the most general kind of calculation is algorithmic, people use anything but algorithmic probability as the gold standard against which other approaches are compared. The "problems" with algorithmic probability (uncomputability, UTM "choice" etc.) seem to be "the dog ate my homework" excuses. No scientific model is required to prove itself to be the best of all possible models relative to a given set of observations in order to be considered the best current model relative to those observations. No "UTM" chosen on the basis of the observations to be modeled is reasonably considered anything but post-hoc theorizing.
- jmuncaster 3y agoBernoulli’s Fallacy by Aubrey Clayton is a good treatment of this subject and where frequentist statistics fails.
- scotty79 3y agoI had trouble grasping difference between frequentist vs bayesian. After googling frantically I found this article which turned out to be very informative about how bayesian analysis of a case might look like vs frequentist one: https://www.redjournal.org/article/S0360-3016(21)03256-9/fulltext https://www.redjournal.org/article/S0360-3016(21)03256-9/ful... I didn't understand much from the part where they use Markov chains to calculate probability of something that is hard to know but the rest illustrates differences really well.
- blamefisher 3y agoThere is a reproduceability crisis in published literature. Most of these study results rely on Frequentist statistics. Many of these studies are just not repeatable, or worse, clearly wrong. Use of Bayesian statistics might have averted this situation.
- therobots927 3y agoI’m really not a fan of Bayesian philosophy. The concept of a prior is kind of preposterous. It’s used consistently to make arguments that basically go “if you assume a prior it proves a probability of zero/1”. The concept of belief is slippery, all that matters is measured probability from data. Anything else is just speculation. In this case, using prior trial data to inflate sample size is entirely counter to the scientific method. You’re basically just reinforcing whatever conclusions previous trials came to. You’re potentially setting yourself up for a string of false positive trial results. If your first 20 trials have a hidden error in experiment design, future studies will continue to exaggerate the effect to an even greater degree as time goes on. I really wish this type of thinking never started in the first place.
- Viliam1234 3y agoThe choice is not between using or not using priors, but between admitting it or denying it. A calculation that "does not use priors" is usually mathematically equivalent to one that does, for some specific prior.
- therobots927 3y agoThe central limit theorem lets us calculate confidence intervals around population statistics regardless of the underlying distribution. If you're talking about working with small datasets, or speculating about the probability of events that have never occurred (n = 0), then just admit that math does not apply and that you're speculating.
- kgwgk 3y agoThe problem with frequentist confidence intervals is that they don't really tell us anything about the probability that the statistic of interest for our population is contained in the interval - regardless of the size of our dataset. Not everyone wants to admit - or even understands - that the % confidence is about how frequently such intervals will contain the true value when the procedure is applied to other populations / datasets.