4 ms·

i have a decent grasp of stats (having read all of casella berger) and i don't understand how you can truly quantify uncertainty since confidence intervals or p

i have a decent grasp of stats (having read all of casella berger) and i don't understand how you can truly quantify uncertainty since confidence intervals or pvalues assume a population distribution. i feel like it's a shell game. given a uniform prior what can you tell me about the uncertainty? and if you've chosen some other prior how can you qualify that decision? this is not even considering that if you take it all the way down to first principles, there isn't even such a thing as a continuous prior or one with uncountable support.

Mooks79 7y ago
This is an area where Bayesian methods really come to the fore - dealing with non-normal, even non-parametric, distributions. That’s not to say they can’t be treated in frequentists stats. I’ve never read that book but perhaps it stops before dealing with how to handle all that stuff. Regarding your specific point about how do you quantify error regarding your chosen prior? The answer in a Bayesian framework lies in things like credibility intervals, posterior distributions, posterior predictive distributions, depending on what you’re actually trying to quantify. Very long story short, the latter allows you to sample from a distribution of predictions - for which you can then form a variety of error estimates (and the distribution doesn’t have to be parametric). That will quantify uncertainty for the model you’ve made using the prior you chose (and the data). But usually we wouldn’t recommend changing your prior just to get lower uncertainty unless you have some rationality behind changing the prior. Priors should really be chosen based on domain knowledge etc etc etc.
teataster 7y ago
Maybe you should read something other than Casella & Berger for Bayesian statistics. The book is great, but the coverage in this topic is not. Something like Bayesian methods for hackers [1], the puppy book [2] are great tutorial style easy to read books. Big boy Gelman [2] is far more mathematical and advanced. I highly recommend you [1] cause it's free, hands on and if you have solid stats will go fast and easy. Anyways, the thing with Bayesian is you need to look at the problems from a different optic and learn new names for the same things you know from classic (Maximum likelihood is now called mean a posteriori and whatnot). If I had to summarize the whole uncertainty thing I would say: Both in classic and Bayesian it holds that a function of random variables yields another random variable. Models are functions of random variables. As such, every point estimate you give has a full distribution behind. In classic this is accounted for with asymptotics and confidence intervals. In classics you have priors all the same, but using defaults and not talking about them. In Bayesian you talk about your priors, and if you have extra information about your inference problem you include it there. Or you abuse the priors to make the model yield results that will make your boss/client happy (I worked as an economics researcher at a big bank, the whole department was Bayesian). Or choose priors that make you custom model actually compute. Or use priors that do regularization (like a classic lasso)... Different priors are different models and you need to do your model checking / selection. I am not so sure what you mean about continuous priors. Any gamma should be continuous. But I don't think that is what you mean. [1] https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers https://github.com/CamDavidsonPilon/Probabilistic-Programmin... [2] https://www.elsevier.com/books/doing-bayesian-data-analysis/kruschke/978-0-12-405888-0 https://www.elsevier.com/books/doing-bayesian-data-analysis/... [3] http://www.stat.columbia.edu/~gelman/book/ http://www.stat.columbia.edu/~gelman/book/

Mooks79 7y ago

This is an area where Bayesian methods really come to the fore - dealing with non-normal, even non-parametric, distributions. That’s not to say they can’t be treated in frequentists stats. I’ve never read that book but perhaps it stops before dealing with how to handle all that stuff. Regarding your specific point about how do you quantify error regarding your chosen prior? The answer in a Bayesian framework lies in things like credibility intervals, posterior distributions, posterior predictive distributions, depending on what you’re actually trying to quantify. Very long story short, the latter allows you to sample from a distribution of predictions - for which you can then form a variety of error estimates (and the distribution doesn’t have to be parametric). That will quantify uncertainty for the model you’ve made using the prior you chose (and the data). But usually we wouldn’t recommend changing your prior just to get lower uncertainty unless you have some rationality behind changing the prior. Priors should really be chosen based on domain knowledge etc etc etc.

teataster 7y ago

Maybe you should read something other than Casella & Berger for Bayesian statistics. The book is great, but the coverage in this topic is not. Something like Bayesian methods for hackers [1], the puppy book [2] are great tutorial style easy to read books. Big boy Gelman [2] is far more mathematical and advanced. I highly recommend you [1] cause it's free, hands on and if you have solid stats will go fast and easy. Anyways, the thing with Bayesian is you need to look at the problems from a different optic and learn new names for the same things you know from classic (Maximum likelihood is now called mean a posteriori and whatnot). If I had to summarize the whole uncertainty thing I would say: Both in classic and Bayesian it holds that a function of random variables yields another random variable. Models are functions of random variables. As such, every point estimate you give has a full distribution behind. In classic this is accounted for with asymptotics and confidence intervals. In classics you have priors all the same, but using defaults and not talking about them. In Bayesian you talk about your priors, and if you have extra information about your inference problem you include it there. Or you abuse the priors to make the model yield results that will make your boss/client happy (I worked as an economics researcher at a big bank, the whole department was Bayesian). Or choose priors that make you custom model actually compute. Or use priors that do regularization (like a classic lasso)... Different priors are different models and you need to do your model checking / selection. I am not so sure what you mean about continuous priors. Any gamma should be continuous. But I don't think that is what you mean. [1] https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers https://github.com/CamDavidsonPilon/Probabilistic-Programmin... [2] https://www.elsevier.com/books/doing-bayesian-data-analysis/kruschke/978-0-12-405888-0 https://www.elsevier.com/books/doing-bayesian-data-analysis/... [3] http://www.stat.columbia.edu/~gelman/book/ http://www.stat.columbia.edu/~gelman/book/