4 ms·

“Non-Bayesian” stats uses bias in the exact same way that Bayesian stats does, because Bayesian stats and frequentist stats are mathematically equivalent. When

“Non-Bayesian” stats uses bias in the exact same way that Bayesian stats does, because Bayesian stats and frequentist stats are mathematically equivalent. When someone thinks they’re not using priors, they’re wrong, they’re usually using a flat prior on the model parameters, but that adds bias just like any other prior! A flat prior on theta is different than a flat prior on log(theta) or some other parametrization, and flat priors are often times the wrong choice, so this notion that Bayesian stats is some “special” type of inference and that there is some way to do inference without bias is just a very large misconception.

CrazyStat 6y ago
> “Non-Bayesian” stats uses bias in the exact same way that Bayesian stats does This is incorrect. Bias has a very specific specific mathematical meaning in statistics--the difference between the expected value of the estimate (under the sampling distribution) and the true value. There are many examples of parameter estimates in classical statistics that have zero bias under that definition. > Bayesian stats and frequentist stats are mathematically equivalent. Also incorrect. Bayesian and frequentist methods focus on different conditional probabilities and can give very divergent results even in simple cases. See e.g. Lindley's paradox [1]. [1] https://en.wikipedia.org/wiki/Lindley%27s%20paradox https://en.wikipedia.org/wiki/Lindley%27s%20paradox
- astrophysician 6y ago
  I appreciate your comments. > Bias has a very specific specific mathematical meaning in statistics--the difference between the expected value of the estimate (under the sampling distribution) and the true value. Right, I'm aware of what bias is, and how it's defined, and while the statement you make is true, it misses the point: regardless of the camp you're in (frequentist or Bayesian), inference involves a prior, and that prior will affect your inferred parameter estimates (it may bias them, it may not, but flat priors do not guarantee unbiased estimates). I agree, under various contrived scenarios, you can show your parameter estimate is unbiased when using a flat prior (yay!) but what happens if you're using the wrong parameterization of your model? A flat prior on \theta is not a flat prior on \log\theta or any other transformation of theta, but if you're a frequentist what do you do about that? If you're not conscious of the prior choice you are making, you can easily introduce bias you don't want even with a flat prior. > There are many examples of parameter estimates in classical statistics that have zero bias under that definition. Elaborate. I assume by "parameter estimates" you mean MAP (or MLE = MAP with flat prior). E[\hat{\theta} - \theta] may be zero with a flat prior, but E[\hat{\log\theta} - \log\theta] wont be, so the bias in your parameter estimate depends on the parameter you really care about. see e.g. [1] for an example where flat priors do actually bias inferences. > Also incorrect. Bayesian and frequentist methods focus on different conditional probabilities and can give very divergent results even in simple cases. See e.g. Lindley's paradox [1]. OK so let me clarify because I agree that my wording is wrong: while I agree with you that there are differences between Bayesian and frequentist statistics, they are philosophical; they answer different questions: ironically the wikipedia article you linked to actually says it pretty well: "Although referred to as a paradox, the differing results from the Bayesian and frequentist approaches can be explained as using them to answer fundamentally different questions, rather than actual disagreement between the two methods." [1]: https://mc-stan.org/users/documentation/case-studies/weakly_informative_shapes.html https://mc-stan.org/users/documentation/case-studies/weakly_... "Although flat priors are often motivated as being “non-informative”, they are actually quite informative and pull the posterior towards extreme values that can bias our inferences."
  - CrazyStat 6y ago
    > I agree, under various contrived scenarios, you can show your parameter estimate is unbiased when using a flat prior (yay!) but what happens if you're using the wrong parameterization of your model? I wouldn't consider estimating, say, the mean length of a population of fish contrived (unbiased estimate: x-bar). Nor would I consider estimating the probability of an event based on observations of the event happening or not happening contrived (unbiased estimate: p-hat = #successes/#trials). These kinds of simple estimation problems and the associated statistical tests account for probably 90% of statistical practice. Dismissing them as contrived is silly. More generally, MLE estimates are always (under regularity conditions) asymptotically unbiased even if not unbiased for a finite sample. This means that the amount of bias decreases to zero as the sample size increases, no matter what the parameterization is. Finally, there is very often a natural parameterization for any given problem. If you're interested in the arithmetic mean of a population, there's no reason to use a log-scale parameterization. Why worry about bias in other parameterizations when you can just use the natural parameterization, where the estimator is unbiased? Again, I don't think such scenarios are contrived: a very large proportion of statistical analyses deal with simple measurements in Euclidian (or very nearly Euclidian--we can typically ignore, for example, relativistic effects) spaces: real world dimensions, time, etc. If you're a Bayesian and very concerned about parameterization effects you can also use a Jeffrey's prior, which is parameterization-invariant. Notably, for the mean of a Normal distribution, the Jeffrey's prior is... the flat prior! > OK so let me clarify because I agree that my wording is wrong: while I agree with you that there are differences between Bayesian and frequentist statistics, they are philosophical; they answer different questions: Yes and no. The Bayesian and frequentist approaches answer different mathematical questions, but they are used by humans to answer the same human questions, such as "do these two populations have the same mean?" > see e.g. [1] for an example where flat priors do actually bias inferences. I don't consider that a good example of flat priors biasing the inference. The posterior with flat prior is diffuse because the data doesn't provide much information; in the lack of much prior or likelihood information, the posterior should be diffuse, so that's a perfectly reasonable result. If you can't stand a diffuse posterior then either collect more data or (carefully!) introduce a more informative prior. The more informative the prior you introduce, the more biased your inference will be; that's fine as long as your prior is chosen carefully. This is not to say that flat priors are never a problem. The setting I'm familiar with where diffuse priors are most dangerous is when doing model comparison--but that issue is specific to Bayesian methods, not frequentist. Indeed this is one of the primary reasons Lindley's paradox arises: the Bayesian model comparison (using marginal likelihoods or Bayes factors) gets tricked by the diffuse prior, while the frequentist model comparison (using null hypothesis testing) does not.

CrazyStat 6y ago

> “Non-Bayesian” stats uses bias in the exact same way that Bayesian stats does This is incorrect. Bias has a very specific specific mathematical meaning in statistics--the difference between the expected value of the estimate (under the sampling distribution) and the true value. There are many examples of parameter estimates in classical statistics that have zero bias under that definition. > Bayesian stats and frequentist stats are mathematically equivalent. Also incorrect. Bayesian and frequentist methods focus on different conditional probabilities and can give very divergent results even in simple cases. See e.g. Lindley's paradox [1]. [1] https://en.wikipedia.org/wiki/Lindley%27s%20paradox https://en.wikipedia.org/wiki/Lindley%27s%20paradox

astrophysician 6y ago
I appreciate your comments. > Bias has a very specific specific mathematical meaning in statistics--the difference between the expected value of the estimate (under the sampling distribution) and the true value. Right, I'm aware of what bias is, and how it's defined, and while the statement you make is true, it misses the point: regardless of the camp you're in (frequentist or Bayesian), inference involves a prior, and that prior will affect your inferred parameter estimates (it may bias them, it may not, but flat priors do not guarantee unbiased estimates). I agree, under various contrived scenarios, you can show your parameter estimate is unbiased when using a flat prior (yay!) but what happens if you're using the wrong parameterization of your model? A flat prior on \theta is not a flat prior on \log\theta or any other transformation of theta, but if you're a frequentist what do you do about that? If you're not conscious of the prior choice you are making, you can easily introduce bias you don't want even with a flat prior. > There are many examples of parameter estimates in classical statistics that have zero bias under that definition. Elaborate. I assume by "parameter estimates" you mean MAP (or MLE = MAP with flat prior). E[\hat{\theta} - \theta] may be zero with a flat prior, but E[\hat{\log\theta} - \log\theta] wont be, so the bias in your parameter estimate depends on the parameter you really care about. see e.g. [1] for an example where flat priors do actually bias inferences. > Also incorrect. Bayesian and frequentist methods focus on different conditional probabilities and can give very divergent results even in simple cases. See e.g. Lindley's paradox [1]. OK so let me clarify because I agree that my wording is wrong: while I agree with you that there are differences between Bayesian and frequentist statistics, they are philosophical; they answer different questions: ironically the wikipedia article you linked to actually says it pretty well: "Although referred to as a paradox, the differing results from the Bayesian and frequentist approaches can be explained as using them to answer fundamentally different questions, rather than actual disagreement between the two methods." [1]: https://mc-stan.org/users/documentation/case-studies/weakly_informative_shapes.html https://mc-stan.org/users/documentation/case-studies/weakly_... "Although flat priors are often motivated as being “non-informative”, they are actually quite informative and pull the posterior towards extreme values that can bias our inferences."
- CrazyStat 6y ago
  > I agree, under various contrived scenarios, you can show your parameter estimate is unbiased when using a flat prior (yay!) but what happens if you're using the wrong parameterization of your model? I wouldn't consider estimating, say, the mean length of a population of fish contrived (unbiased estimate: x-bar). Nor would I consider estimating the probability of an event based on observations of the event happening or not happening contrived (unbiased estimate: p-hat = #successes/#trials). These kinds of simple estimation problems and the associated statistical tests account for probably 90% of statistical practice. Dismissing them as contrived is silly. More generally, MLE estimates are always (under regularity conditions) asymptotically unbiased even if not unbiased for a finite sample. This means that the amount of bias decreases to zero as the sample size increases, no matter what the parameterization is. Finally, there is very often a natural parameterization for any given problem. If you're interested in the arithmetic mean of a population, there's no reason to use a log-scale parameterization. Why worry about bias in other parameterizations when you can just use the natural parameterization, where the estimator is unbiased? Again, I don't think such scenarios are contrived: a very large proportion of statistical analyses deal with simple measurements in Euclidian (or very nearly Euclidian--we can typically ignore, for example, relativistic effects) spaces: real world dimensions, time, etc. If you're a Bayesian and very concerned about parameterization effects you can also use a Jeffrey's prior, which is parameterization-invariant. Notably, for the mean of a Normal distribution, the Jeffrey's prior is... the flat prior! > OK so let me clarify because I agree that my wording is wrong: while I agree with you that there are differences between Bayesian and frequentist statistics, they are philosophical; they answer different questions: Yes and no. The Bayesian and frequentist approaches answer different mathematical questions, but they are used by humans to answer the same human questions, such as "do these two populations have the same mean?" > see e.g. [1] for an example where flat priors do actually bias inferences. I don't consider that a good example of flat priors biasing the inference. The posterior with flat prior is diffuse because the data doesn't provide much information; in the lack of much prior or likelihood information, the posterior should be diffuse, so that's a perfectly reasonable result. If you can't stand a diffuse posterior then either collect more data or (carefully!) introduce a more informative prior. The more informative the prior you introduce, the more biased your inference will be; that's fine as long as your prior is chosen carefully. This is not to say that flat priors are never a problem. The setting I'm familiar with where diffuse priors are most dangerous is when doing model comparison--but that issue is specific to Bayesian methods, not frequentist. Indeed this is one of the primary reasons Lindley's paradox arises: the Bayesian model comparison (using marginal likelihoods or Bayes factors) gets tricked by the diffuse prior, while the frequentist model comparison (using null hypothesis testing) does not.