5 ms·

> I’m sure you’re asking: “why allow your parameters to be biased?” Good question. The most straightforward answer is that there is a bias-variance trade-off. T

> I’m sure you’re asking: “why allow your parameters to be biased?” Good question. The most straightforward answer is that there is a bias-variance trade-off. The Wikipedia article does a good job both illustrating and explaining it. For β-hat purposes, the notion of allowing any bias is crazy. For y-hat purposes, adding a little bias in exchange for a huge reduction in variance can improve the predictive power of your model. I'm going to push back on this. The author seems to understand the bias-variance tradeoff as applying primarily to y-hat, and allows that if you are primarily interested in y-hat then it can make sense to make that tradeoff (introduce bias in exchange for lower variance). But the bias-variance tradeoff is more general than that. There's also a bias-variance tradeoff in beta-hat, and you can make a similar decision there to introduce some bias in beta-hat in exchange for lower variance, lowering the overall mean square error. There's nothing crazy about this. The entire field[1] of Bayesian statistics does this every day--Bayesian priors introduce bias in the parameters, with the benefit of decreasing variance. Bayesians use these biased parameter estimates without any problems. Classical (non-Bayesian) statistics has tended to focus heavily on unbiased models. I suspect this is largely because restricting the class of models you're looking at to unbiased models allows you to prove a lot of interesting results. For example, if you restrict yourself to linear unbiased models, you can identify one single `best` (i.e. lowest variance) estimator. As soon as you allow bias you can't do that anymore. [1] Except empirical Bayes, which is a dark art.

ethelward 6y ago
> [1] Except empirical Bayes, which is a dark art. As a not-an-expert-in-stats, why would you say that? Is not empirical Bayes basically the same thing, but with priors stemming from the known data?
- CrazyStat 6y ago
  > Is not empirical Bayes basically the same thing, but with priors stemming from the known data? Yes. The problem is that either you end up using the data twice (once in the prior and once in the likelihood) or you have to choose how to split the data between the prior and likelihood, which can lead to other problems (particularly if you want to compare different models).
  - ethelward 6y ago
    Thanks!
astrophysician 6y ago
“Non-Bayesian” stats uses bias in the exact same way that Bayesian stats does, because Bayesian stats and frequentist stats are mathematically equivalent. When someone thinks they’re not using priors, they’re wrong, they’re usually using a flat prior on the model parameters, but that adds bias just like any other prior! A flat prior on theta is different than a flat prior on log(theta) or some other parametrization, and flat priors are often times the wrong choice, so this notion that Bayesian stats is some “special” type of inference and that there is some way to do inference without bias is just a very large misconception.
- CrazyStat 6y ago
  > “Non-Bayesian” stats uses bias in the exact same way that Bayesian stats does This is incorrect. Bias has a very specific specific mathematical meaning in statistics--the difference between the expected value of the estimate (under the sampling distribution) and the true value. There are many examples of parameter estimates in classical statistics that have zero bias under that definition. > Bayesian stats and frequentist stats are mathematically equivalent. Also incorrect. Bayesian and frequentist methods focus on different conditional probabilities and can give very divergent results even in simple cases. See e.g. Lindley's paradox [1]. [1] https://en.wikipedia.org/wiki/Lindley%27s%20paradox https://en.wikipedia.org/wiki/Lindley%27s%20paradox
  - astrophysician 6y ago
    I appreciate your comments. > Bias has a very specific specific mathematical meaning in statistics--the difference between the expected value of the estimate (under the sampling distribution) and the true value. Right, I'm aware of what bias is, and how it's defined, and while the statement you make is true, it misses the point: regardless of the camp you're in (frequentist or Bayesian), inference involves a prior, and that prior will affect your inferred parameter estimates (it may bias them, it may not, but flat priors do not guarantee unbiased estimates). I agree, under various contrived scenarios, you can show your parameter estimate is unbiased when using a flat prior (yay!) but what happens if you're using the wrong parameterization of your model? A flat prior on \theta is not a flat prior on \log\theta or any other transformation of theta, but if you're a frequentist what do you do about that? If you're not conscious of the prior choice you are making, you can easily introduce bias you don't want even with a flat prior. > There are many examples of parameter estimates in classical statistics that have zero bias under that definition. Elaborate. I assume by "parameter estimates" you mean MAP (or MLE = MAP with flat prior). E[\hat{\theta} - \theta] may be zero with a flat prior, but E[\hat{\log\theta} - \log\theta] wont be, so the bias in your parameter estimate depends on the parameter you really care about. see e.g. [1] for an example where flat priors do actually bias inferences. > Also incorrect. Bayesian and frequentist methods focus on different conditional probabilities and can give very divergent results even in simple cases. See e.g. Lindley's paradox [1]. OK so let me clarify because I agree that my wording is wrong: while I agree with you that there are differences between Bayesian and frequentist statistics, they are philosophical; they answer different questions: ironically the wikipedia article you linked to actually says it pretty well: "Although referred to as a paradox, the differing results from the Bayesian and frequentist approaches can be explained as using them to answer fundamentally different questions, rather than actual disagreement between the two methods." [1]: https://mc-stan.org/users/documentation/case-studies/weakly_informative_shapes.html https://mc-stan.org/users/documentation/case-studies/weakly_... "Although flat priors are often motivated as being “non-informative”, they are actually quite informative and pull the posterior towards extreme values that can bias our inferences."
curiousgal 6y ago
Limiting the number of models to choose from is not that useful from a practical point of view. For instance, I am yet to come across any practical use of the Vapnik–Chervonenkis dimension.
- CrazyStat 6y ago
  If you want to prove a result (for example the existence of a unique minimum-variance estimator) for a class of models, but can only prove it by restricting the class, then restricting the class is useful for that purpose. It may not be useful in applications, other than if you want assurances provided by results that have been proven about the class of model you're using.

ethelward 6y ago

> [1] Except empirical Bayes, which is a dark art. As a not-an-expert-in-stats, why would you say that? Is not empirical Bayes basically the same thing, but with priors stemming from the known data?

CrazyStat 6y ago
> Is not empirical Bayes basically the same thing, but with priors stemming from the known data? Yes. The problem is that either you end up using the data twice (once in the prior and once in the likelihood) or you have to choose how to split the data between the prior and likelihood, which can lead to other problems (particularly if you want to compare different models).
- ethelward 6y ago
  Thanks!

astrophysician 6y ago

“Non-Bayesian” stats uses bias in the exact same way that Bayesian stats does, because Bayesian stats and frequentist stats are mathematically equivalent. When someone thinks they’re not using priors, they’re wrong, they’re usually using a flat prior on the model parameters, but that adds bias just like any other prior! A flat prior on theta is different than a flat prior on log(theta) or some other parametrization, and flat priors are often times the wrong choice, so this notion that Bayesian stats is some “special” type of inference and that there is some way to do inference without bias is just a very large misconception.

CrazyStat 6y ago
> “Non-Bayesian” stats uses bias in the exact same way that Bayesian stats does This is incorrect. Bias has a very specific specific mathematical meaning in statistics--the difference between the expected value of the estimate (under the sampling distribution) and the true value. There are many examples of parameter estimates in classical statistics that have zero bias under that definition. > Bayesian stats and frequentist stats are mathematically equivalent. Also incorrect. Bayesian and frequentist methods focus on different conditional probabilities and can give very divergent results even in simple cases. See e.g. Lindley's paradox [1]. [1] https://en.wikipedia.org/wiki/Lindley%27s%20paradox https://en.wikipedia.org/wiki/Lindley%27s%20paradox
- astrophysician 6y ago
  I appreciate your comments. > Bias has a very specific specific mathematical meaning in statistics--the difference between the expected value of the estimate (under the sampling distribution) and the true value. Right, I'm aware of what bias is, and how it's defined, and while the statement you make is true, it misses the point: regardless of the camp you're in (frequentist or Bayesian), inference involves a prior, and that prior will affect your inferred parameter estimates (it may bias them, it may not, but flat priors do not guarantee unbiased estimates). I agree, under various contrived scenarios, you can show your parameter estimate is unbiased when using a flat prior (yay!) but what happens if you're using the wrong parameterization of your model? A flat prior on \theta is not a flat prior on \log\theta or any other transformation of theta, but if you're a frequentist what do you do about that? If you're not conscious of the prior choice you are making, you can easily introduce bias you don't want even with a flat prior. > There are many examples of parameter estimates in classical statistics that have zero bias under that definition. Elaborate. I assume by "parameter estimates" you mean MAP (or MLE = MAP with flat prior). E[\hat{\theta} - \theta] may be zero with a flat prior, but E[\hat{\log\theta} - \log\theta] wont be, so the bias in your parameter estimate depends on the parameter you really care about. see e.g. [1] for an example where flat priors do actually bias inferences. > Also incorrect. Bayesian and frequentist methods focus on different conditional probabilities and can give very divergent results even in simple cases. See e.g. Lindley's paradox [1]. OK so let me clarify because I agree that my wording is wrong: while I agree with you that there are differences between Bayesian and frequentist statistics, they are philosophical; they answer different questions: ironically the wikipedia article you linked to actually says it pretty well: "Although referred to as a paradox, the differing results from the Bayesian and frequentist approaches can be explained as using them to answer fundamentally different questions, rather than actual disagreement between the two methods." [1]: https://mc-stan.org/users/documentation/case-studies/weakly_informative_shapes.html https://mc-stan.org/users/documentation/case-studies/weakly_... "Although flat priors are often motivated as being “non-informative”, they are actually quite informative and pull the posterior towards extreme values that can bias our inferences."

curiousgal 6y ago

Limiting the number of models to choose from is not that useful from a practical point of view. For instance, I am yet to come across any practical use of the Vapnik–Chervonenkis dimension.

CrazyStat 6y ago
If you want to prove a result (for example the existence of a unique minimum-variance estimator) for a class of models, but can only prove it by restricting the class, then restricting the class is useful for that purpose. It may not be useful in applications, other than if you want assurances provided by results that have been proven about the class of model you're using.