8 ms·

Reminder to all the HN/LW/etc Bayesians here that this is a frequentist analysis and therefore politically unacceptable.

jmount 13y ago
true
- sanskritabelt 13y ago
  It's a good post, btw.
RA_Fisher 13y ago
What does this get us? Okay, so we can more accurately calculate sample sizes, but what about power relative to the "power" of using a Beta prior in the Bayesian framework? I use Bayesian methods simply to restrict the test space. It's also relatively simple (computer is doing the math, essentially).
- jmount 13y ago
  If somebody proposes a level of improvement they expect in an A/B test (and you have a good prior estimate of the untreated A-rate) they you quickly get specific sample sizes to route to A and B. You also have the form of expression so you can see what is expensive (measuring small differences) and what is cheap (driving up confidence). Frankly a Bayesian article that makes a similar calculation would be at least as good an article (as it would have a much less clumsy treatment of what are you expecting to measure as a distribution prior instead of a point estimate). But I think the merit is how fast you can get from start to finish in such calculations. Edit: I've added some discussion on what you would expect in the Bayesian case as an comment on the blog.
adamtj 13y ago
I'm not much of a statistician. Can anybody explain to me why Bayesian vs Frequentist is such a political or religious issue? It seems to me they must either describe the same underlying reality correctly, though differently, or else one is objectively wrong. If they are both correct but different, can we not just characterize the differences and use whichever is most appropriate? It it just a matter of taste with differing opinions on the correctness (or morality?) of the guessing that's necessary for Bayesian analysis? Can we not just figure out the error as a function of the wrongness of the guess and be done with it? Are there guesses so wrong that no amount of data is sufficient for Bayesian and Frequentist analyses to converge?
- jmount 13y ago
  A lot of the calculations (including the ability to use Bayes law) is common to Frequentists and Bayesians. Efron says the difference is Frequentists assume the quantity to be modeled is unknown but has a single fixed value (so you can't make probability statements about it) and the measurements have some possible variability. Bayesians model the unknown with an explicit prior distribution for the unknown quantity to be inferred and assume the measurements are fixed. So each school means something different when they state a probability (Frequentist variation over data draws, Bayesian variation over the unknown to be measured). Gelman defines Bayesian methods as the proper setting up of a generative model and computing joint probabilities of parameters and data. In his view Frequentist don't compute joint probabilities but at best conditional probabilities (which assume out modeling likely values for the unknown quantity to be inferred). The joke of the comment is: both sides tend to snipe on the other (as neither framework looks good if you assume the other). My partner has a nice article on the topic: http://www.win-vector.com/blog/2013/05/bayesian-and-frequentist-approaches-ask-the-right-question/ http://www.win-vector.com/blog/2013/05/bayesian-and-frequent...
  - adamtj 13y ago
    Thanks! I got that it was a joke, I've just never understood why it was funny. The "debate" has always seemed to me like Quantumist physicists arguing with Classicalists about which method of computing a cannonball trajectory is correct. I mean, clearly the Classicalist interpretation overestimates the effectiveness of cannons by failing to account for the chance that the ball may quantum-tunnel through its target, leaving it undamaged. Your comment and that article really help convince me that there exist people who actually understand what's going on.
- Fomite 13y ago
  Bayesian priors aren't necessarily formed by "Guessing", and if we do advance the notion that they are "Guesses", Frequentist statistics has its own set of them.
- moultano 13y ago
  Frequentists and Bayesians both agree on all of the math. The difference lies in the interpretation of what the math means, and how useful different types of math are. A rough summary: Frequentist methods tend to try to characterize the question of interest in terms of a single measurable statistic. This is particularly useful in circumstances where memory/cpu is tight (i.e. every single circumstance before the last few decades.) This necessarily means though that these techniques lose much of the information in the data. They also try to make very few a priori assumptions about the underlying data. The bayesian critique of this is that the assumptions are still there, just hidden, and as a result are much more likely to be wrong. Bayesian methods tend to try to characterize the question of interest in terms of the entire distribution of outcomes. This causes there to be little information lost between the data and the conclusions, but is often computationally intractable in common circumstances. Bayesians tend to make strong a priori assumptions about the form the data will take, and to ameliorate this they make their assumptions very explicit in the model. The Frequentist critique of this is that every time you read a bayesian piece of analysis, you have to be very careful to determine whether the outcome described is merely the assumed prior + noise. Both cultures provide essential mathematical tools for any statistician's toolbox, and it's often just a matter of taste which get used. Hence the bike-shedding.
diminoten 13y ago
Question for all the HN/LW/etc Bayesians here: how do you assign accurate probabilities to immeasurable events? Don't you find yourself with a false sense of certainty when you use more sig-figs than you have access to? I get how bayesian probability works mathematically, but doesn't it rely on having accurate prior measurements for the likelihood of an outcome? That is, working from prior probabilities only becomes useful and accurate if prior probabilities are accurate to begin with.
- RA_Fisher 13y ago
  With a ton of data like most start-ups have the effect of the priors generally wash away. That is the likelihood (of the data) generally dominates. My results are generally very robust to choice of prior parameters (when the type of distribution is well known like Beta for Bernoulli RVs).
  - diminoten 13y ago
    I have literally no idea what you wrote here. Start-ups? Priors? Beta for Bernoulli RVs? "That is the likelihood generally dominates"? "My results are generally very robust to choice of prior parameters"? Neither of those sentences are comprehensible.
    - yummyfajitas 13y ago
      I have a blog post from a while back which explains in more detail everything in RA_Fisher's comment: http://www.chrisstucchio.com/blog/2013/bayesian_analysis_conversion_rates.html http://www.chrisstucchio.com/blog/2013/bayesian_analysis_con...
      - diminoten 13y ago
        Yeah, this is mathematical application of bayesian analysis, which is a completely valid and proven way to do analysis of actual data. I'm talking more to the folks over on LW who say things such as, "It's 17% likely that my actions will cause harm, so I'll act./won't act."

jmount 13y ago

true

sanskritabelt 13y ago
It's a good post, btw.

RA_Fisher 13y ago

What does this get us? Okay, so we can more accurately calculate sample sizes, but what about power relative to the "power" of using a Beta prior in the Bayesian framework? I use Bayesian methods simply to restrict the test space. It's also relatively simple (computer is doing the math, essentially).

jmount 13y ago
If somebody proposes a level of improvement they expect in an A/B test (and you have a good prior estimate of the untreated A-rate) they you quickly get specific sample sizes to route to A and B. You also have the form of expression so you can see what is expensive (measuring small differences) and what is cheap (driving up confidence). Frankly a Bayesian article that makes a similar calculation would be at least as good an article (as it would have a much less clumsy treatment of what are you expecting to measure as a distribution prior instead of a point estimate). But I think the merit is how fast you can get from start to finish in such calculations. Edit: I've added some discussion on what you would expect in the Bayesian case as an comment on the blog.

adamtj 13y ago

I'm not much of a statistician. Can anybody explain to me why Bayesian vs Frequentist is such a political or religious issue? It seems to me they must either describe the same underlying reality correctly, though differently, or else one is objectively wrong. If they are both correct but different, can we not just characterize the differences and use whichever is most appropriate? It it just a matter of taste with differing opinions on the correctness (or morality?) of the guessing that's necessary for Bayesian analysis? Can we not just figure out the error as a function of the wrongness of the guess and be done with it? Are there guesses so wrong that no amount of data is sufficient for Bayesian and Frequentist analyses to converge?

jmount 13y ago
A lot of the calculations (including the ability to use Bayes law) is common to Frequentists and Bayesians. Efron says the difference is Frequentists assume the quantity to be modeled is unknown but has a single fixed value (so you can't make probability statements about it) and the measurements have some possible variability. Bayesians model the unknown with an explicit prior distribution for the unknown quantity to be inferred and assume the measurements are fixed. So each school means something different when they state a probability (Frequentist variation over data draws, Bayesian variation over the unknown to be measured). Gelman defines Bayesian methods as the proper setting up of a generative model and computing joint probabilities of parameters and data. In his view Frequentist don't compute joint probabilities but at best conditional probabilities (which assume out modeling likely values for the unknown quantity to be inferred). The joke of the comment is: both sides tend to snipe on the other (as neither framework looks good if you assume the other). My partner has a nice article on the topic: http://www.win-vector.com/blog/2013/05/bayesian-and-frequentist-approaches-ask-the-right-question/ http://www.win-vector.com/blog/2013/05/bayesian-and-frequent...
- adamtj 13y ago
  Thanks! I got that it was a joke, I've just never understood why it was funny. The "debate" has always seemed to me like Quantumist physicists arguing with Classicalists about which method of computing a cannonball trajectory is correct. I mean, clearly the Classicalist interpretation overestimates the effectiveness of cannons by failing to account for the chance that the ball may quantum-tunnel through its target, leaving it undamaged. Your comment and that article really help convince me that there exist people who actually understand what's going on.
Fomite 13y ago
Bayesian priors aren't necessarily formed by "Guessing", and if we do advance the notion that they are "Guesses", Frequentist statistics has its own set of them.
moultano 13y ago
Frequentists and Bayesians both agree on all of the math. The difference lies in the interpretation of what the math means, and how useful different types of math are. A rough summary: Frequentist methods tend to try to characterize the question of interest in terms of a single measurable statistic. This is particularly useful in circumstances where memory/cpu is tight (i.e. every single circumstance before the last few decades.) This necessarily means though that these techniques lose much of the information in the data. They also try to make very few a priori assumptions about the underlying data. The bayesian critique of this is that the assumptions are still there, just hidden, and as a result are much more likely to be wrong. Bayesian methods tend to try to characterize the question of interest in terms of the entire distribution of outcomes. This causes there to be little information lost between the data and the conclusions, but is often computationally intractable in common circumstances. Bayesians tend to make strong a priori assumptions about the form the data will take, and to ameliorate this they make their assumptions very explicit in the model. The Frequentist critique of this is that every time you read a bayesian piece of analysis, you have to be very careful to determine whether the outcome described is merely the assumed prior + noise. Both cultures provide essential mathematical tools for any statistician's toolbox, and it's often just a matter of taste which get used. Hence the bike-shedding.

diminoten 13y ago

Question for all the HN/LW/etc Bayesians here: how do you assign accurate probabilities to immeasurable events? Don't you find yourself with a false sense of certainty when you use more sig-figs than you have access to? I get how bayesian probability works mathematically, but doesn't it rely on having accurate prior measurements for the likelihood of an outcome? That is, working from prior probabilities only becomes useful and accurate if prior probabilities are accurate to begin with.

RA_Fisher 13y ago
With a ton of data like most start-ups have the effect of the priors generally wash away. That is the likelihood (of the data) generally dominates. My results are generally very robust to choice of prior parameters (when the type of distribution is well known like Beta for Bernoulli RVs).
- diminoten 13y ago
  I have literally no idea what you wrote here. Start-ups? Priors? Beta for Bernoulli RVs? "That is the likelihood generally dominates"? "My results are generally very robust to choice of prior parameters"? Neither of those sentences are comprehensible.
  - yummyfajitas 13y ago
    I have a blog post from a while back which explains in more detail everything in RA_Fisher's comment: http://www.chrisstucchio.com/blog/2013/bayesian_analysis_conversion_rates.html http://www.chrisstucchio.com/blog/2013/bayesian_analysis_con...
    - diminoten 13y ago
      Yeah, this is mathematical application of bayesian analysis, which is a completely valid and proven way to do analysis of actual data. I'm talking more to the folks over on LW who say things such as, "It's 17% likely that my actions will cause harm, so I'll act./won't act."