6 ms·

Empirical Bayes for multiple sample sizes

104 points by csaid81 9y ago

nl 9y ago
That definition of symbols! So good!
- nerdponx 9y ago
  I know! I love when math books and papers do that.
robterrin 9y ago
Stan is great! Glad to see it on HN. Nice write up too.
phillc73 9y ago
Although tangentially linked to in the article, David Robinson's Introduction to Empirical Bayes[1] is also an excellent resource. It deals primarily with beta-binomial distributions. [1] http://varianceexplained.org/r/empirical-bayes-book/ http://varianceexplained.org/r/empirical-bayes-book/
- csaid81 9y ago
  It's an excellent blog post, although it's worth emphasizing that it is designed for the binomial case, where you wish to compute the fraction of occurrences within some events, such as batting averages. For continuous variables, however, it makes more sense to use one of the methods described in the original post. TL;DR: One blog post is for Rotten Tomatoes and the other is for Metacritic.
  - phillc73 9y ago
    Absolutely, and thanks for better defining the distinction. I really just wanted to point out another solid Empirical Bayes resource, as there's not that many about. Yours and David's make a good combination covering different cases.
pps43 9y ago
Technical term is overkill. Just use https://en.wikipedia.org/wiki/Bayesian_average https://en.wikipedia.org/wiki/Bayesian_average
- apathy 9y ago
  Thank you. As a statistician, the fact that mixed effects models (e.g. does this rater tend to rate high?) are overlooked is, IMHO, a death sentence. Too much nomenclature, too early (link to the table within the text, please, and omit needless words), and with too little attention paid to the value of an external citation. Also, MCMC for ratings? Surely you jest. If the author had touched on mixed models, then maybe it would make sense. But given the sample sizes involved here, and the noise in the variance estimates, I recommend that the author investigate mixed models tout suite if they do in fact care about the sources of shared and unshared effects on variance. Because that is what mixed models do.
  - csaid81 9y ago
    Author here. Please see the section on mixed models in my post. As I mentioned there, I would love if an expert could expand on the relationship between mixed effects and Empirical Bayes. Regarding MCMC, one of the things I try to emphasize throughout the post is that the best solution depends on your needs (for example if you want a full posterior). In fact, most of the post is devoted to quick and simple methods -- not MCMC -- because they are good enough for most purposes. I welcome your feedback though on how I could make this point clearer.
    - apathy 9y ago
      > Author here. Alright, I'll put on my Reviewer Number 3 hat and say that I learned some neat things from your work, including that the National Swine Improvement Federation. I'll try and do a halfway decent job here. > I would love if an expert could expand on the relationship between mixed effects and Empirical Bayes. A real expert? Here you go: http://statweb.stanford.edu/~ckirby/brad/LSI/monograph_CUP.pdf http://statweb.stanford.edu/~ckirby/brad/LSI/monograph_CUP.p... Read it, all of it, but particularly chapter 1, section 2.5, and chapters 8, 10, and 11. Why does testing, effect size estimation, and high-dimensional analysis have anything to do with anything? Because... 1) independence is largely a myth 2) you are likely to have multiple ratings per reviewer on your site, whether your generating distribution is nearly-continuous (0-10, mean-centered) or discrete (0/1, A/B/C). If you discard this, you are throwing away an enormous amount of information, and failing utterly to understand why a person would estimate not just the variance but the covariance even for a univariate response. The second point is the one that matters. Also, "empirical Bayes" is in modern parlance equivalent to "Bayes". What's the alternative? "Conjectural Bayes"? (Maybe I should quit while I'm ahead, pure frequentists may be lurking somewhere) > I welcome your feedback though on how I could make this point clearer. For starters, edit. Your post is too damned long. Think about where you are getting diminishing returns and why. Is there ever a realistic situation where your ratings site would not keep track of who submitted the rating? (It's certainly not going to be an unbiased sample, if so; the ballot box will get stuffed) So if you have to keep track of who's voting, you automatically have information to decompose the covariance matrix, and everything else logically follows. A univariate response with a multivariate predictor (say, rating ~ movie*rater) can have multiple sources of variance, and estimating these from small samples is hard. When you use a James-Stein estimator, you trade variance for bias. You're shrinking towards movie-specific variance estimates, but you almost certainly have enough information to shrink towards movie-centric and rater-centric estimates of fixed and random effects, tempered by the number of ratings per movie and the number of ratings per rater. (Obviously you should not have more than one rating per movie per rater, else your sample cannot be unbiased). I think you will return to this and write a much crisper, more concise, and more useful summary once this sinks in. I could be wrong. But you'll have learned something deeply useful even if I am. I do not think you can lose by it.

104 points by csaid81 9y ago

nl 9y ago

That definition of symbols! So good!

nerdponx 9y ago
I know! I love when math books and papers do that.

robterrin 9y ago

Stan is great! Glad to see it on HN. Nice write up too.

phillc73 9y ago

Although tangentially linked to in the article, David Robinson's Introduction to Empirical Bayes[1] is also an excellent resource. It deals primarily with beta-binomial distributions. [1] http://varianceexplained.org/r/empirical-bayes-book/ http://varianceexplained.org/r/empirical-bayes-book/

csaid81 9y ago
It's an excellent blog post, although it's worth emphasizing that it is designed for the binomial case, where you wish to compute the fraction of occurrences within some events, such as batting averages. For continuous variables, however, it makes more sense to use one of the methods described in the original post. TL;DR: One blog post is for Rotten Tomatoes and the other is for Metacritic.
- phillc73 9y ago
  Absolutely, and thanks for better defining the distinction. I really just wanted to point out another solid Empirical Bayes resource, as there's not that many about. Yours and David's make a good combination covering different cases.

pps43 9y ago

Technical term is overkill. Just use https://en.wikipedia.org/wiki/Bayesian_average https://en.wikipedia.org/wiki/Bayesian_average

apathy 9y ago
Thank you. As a statistician, the fact that mixed effects models (e.g. does this rater tend to rate high?) are overlooked is, IMHO, a death sentence. Too much nomenclature, too early (link to the table within the text, please, and omit needless words), and with too little attention paid to the value of an external citation. Also, MCMC for ratings? Surely you jest. If the author had touched on mixed models, then maybe it would make sense. But given the sample sizes involved here, and the noise in the variance estimates, I recommend that the author investigate mixed models tout suite if they do in fact care about the sources of shared and unshared effects on variance. Because that is what mixed models do.
- csaid81 9y ago
  Author here. Please see the section on mixed models in my post. As I mentioned there, I would love if an expert could expand on the relationship between mixed effects and Empirical Bayes. Regarding MCMC, one of the things I try to emphasize throughout the post is that the best solution depends on your needs (for example if you want a full posterior). In fact, most of the post is devoted to quick and simple methods -- not MCMC -- because they are good enough for most purposes. I welcome your feedback though on how I could make this point clearer.
  - apathy 9y ago
    > Author here. Alright, I'll put on my Reviewer Number 3 hat and say that I learned some neat things from your work, including that the National Swine Improvement Federation. I'll try and do a halfway decent job here. > I would love if an expert could expand on the relationship between mixed effects and Empirical Bayes. A real expert? Here you go: http://statweb.stanford.edu/~ckirby/brad/LSI/monograph_CUP.pdf http://statweb.stanford.edu/~ckirby/brad/LSI/monograph_CUP.p... Read it, all of it, but particularly chapter 1, section 2.5, and chapters 8, 10, and 11. Why does testing, effect size estimation, and high-dimensional analysis have anything to do with anything? Because... 1) independence is largely a myth 2) you are likely to have multiple ratings per reviewer on your site, whether your generating distribution is nearly-continuous (0-10, mean-centered) or discrete (0/1, A/B/C). If you discard this, you are throwing away an enormous amount of information, and failing utterly to understand why a person would estimate not just the variance but the covariance even for a univariate response. The second point is the one that matters. Also, "empirical Bayes" is in modern parlance equivalent to "Bayes". What's the alternative? "Conjectural Bayes"? (Maybe I should quit while I'm ahead, pure frequentists may be lurking somewhere) > I welcome your feedback though on how I could make this point clearer. For starters, edit. Your post is too damned long. Think about where you are getting diminishing returns and why. Is there ever a realistic situation where your ratings site would not keep track of who submitted the rating? (It's certainly not going to be an unbiased sample, if so; the ballot box will get stuffed) So if you have to keep track of who's voting, you automatically have information to decompose the covariance matrix, and everything else logically follows. A univariate response with a multivariate predictor (say, rating ~ movie*rater) can have multiple sources of variance, and estimating these from small samples is hard. When you use a James-Stein estimator, you trade variance for bias. You're shrinking towards movie-specific variance estimates, but you almost certainly have enough information to shrink towards movie-centric and rater-centric estimates of fixed and random effects, tempered by the number of ratings per movie and the number of ratings per rater. (Obviously you should not have more than one rating per movie per rater, else your sample cannot be unbiased). I think you will return to this and write a much crisper, more concise, and more useful summary once this sinks in. I could be wrong. But you'll have learned something deeply useful even if I am. I do not think you can lose by it.