5 ms·

You could also go one step further on the Bayesian path and infer even alpha from the data on your site, and introduce a loss function on your ordering. Or you

by jules 15y ago

You could also go one step further on the Bayesian path and infer even alpha from the data on your site, and introduce a loss function on your ordering. Or you could do a semi-frequentist thing and simplify your math by using MAP estimates to rank. Basically instead of score = #pos/(#pos + #neg), it becomes score = (#pos+x)/(#pos+x + #neg+y), where you choose x and y to suit your needs. You could choose x/y in proportion to the average number of up/down votes on your site or you could even choose x/y in proportion to the average number of up/down votes of the author of the post. That would rank posts of trolls lower than posts of good users. By varying x and y you can tweak the strength of this effect. You can interpret this as giving each item by default x upvotes and y downvotes. This certainly works much better than the formula in the article. For example if a post has 1 upvote and 2 downvotes, his formula will say that should be ranked lower than a post with 1000 upvotes and 2000 downvotes (because he's using the lower bound of the confidence interval). Obviously that's bad because while the first post could be a good one, we know for certain that the second one isn't. In general his method will rank posts with a low number of votes very low, even compared to posts with a high number of downvotes.

NathanRice 15y ago
Absolutely! I just wrote a reply where I alluded to that, unfortunately I didn't refresh and see this post or I would have just plugged you instead. The benefit of the Bayesian treatment here that I want to drill down on is how natural it is to adjust the prior to capture your beliefs about how items should be perceived in the presence of incomplete information. The frequentist approach is fine, but it does not provide such a pleasant, intuitive knob to tune.
- jules 15y ago
  Well, it depends on how you look at it. The Bayesian MAP approach is basically the same as the frequentist approach with made up data. Instead of using the maximum likelihood estimator p = pos/(pos+neg) you pretend that each new post already has some up/down votes by default, and then you use simply p = pos/(pos+neg). Seems like an even more intuitive explanation of the Bayesian knob to me! Rather than an abstract "alpha" to a "Dirichlet prior", you get something concrete (the number of made up votes). And you get a simple formula, which some would find desirable. But I agree that the Bayesian approach is conceptually much cleaner. IMO the frequentist approach is just computational corner cutting for when the math in the Bayesian approach gets too involved, which is sometimes useful. What's nice about the Bayesian approach is that you state your assumptions and then it's just turning the math machinery. In contrast, in the frequentist approach the assumptions are interwoven and hidden in arbitrary choices in how the math is done (And then they claim that Bayesians are subjective! It's just that Bayesians admit that they are subjective. Frequentists try to hide the fact that they are more subjective in the math). The not so nice thing is that turning the math machinery is not always so easy and does not always produce fast algorithms. That's where maximum likelihood and friends come in, but I'd view them as an approximation to Bayesian methods.
  - NathanRice 15y ago
    Yes, the explanation of "imaginary votes" is by far the simplest way of thinking about the situation. Having mathematical formalism and a rigorously studied methods makes me feel OK about doing it though, otherwise I would be very uncomfortable with the approach :) I love Bayesian statistics from a conceptual point of view, but the ease with which one ventures into the land of analytic intractability kind of puts me off more complex models. MCMC is such a clumsy tool (in addition to taking forever); variational methods look interesting to me but I don't really feel they are quite there yet.
Eliezer 15y ago
I read the article, went "The ^&#$?" and came here to post this, more or less. It's amazing the $&#! people will get up to when they don't know Bayesian statistics.
- moultano 15y ago
  That really isn't fair to what this is doing. If a bayesian wanted to lower bound the distribution, they would be using exactly the same formula, but with an explicit prior. The typical bayesian solution to this of the dirichlet prior has a significantly different convergence rate, 1/n vs 1/sqrt(n) so the resulting ordering is substantially different as well, and I would argue worse for this application. The method of setting a pessimistic prior cares much less about the number of votes for a popular item than does the method of lower bounding the distribution.
  - Eliezer 15y ago
    If you think most comments are bad, and you have an explicit prior that reflects this, then your ranking will update to "Hey, the comment was actually good!" at the optimal speed - no more, no less - given the rate of incoming ratings. Why would any other rate of updating be better?
    - moultano 15y ago
      Because there's a high variance on your estimate, so in a lot of circumstances just by variance you'll show users bad comments on top most of the time, rather than showing them the good comment that you are certain of. In this context, the pessimistic prior is more of a hack than the confidence interval, because your mental model isn't that most comments suck. Your mental model is that most of them are ok. The pessimistic prior is just there to say, if 1000 people confirm that a comment is ok, we should show that over a comment for which we don't know whether it's ok or not yet, and this reasoning is much better modeled by a lower bound.

by jules 15y ago

NathanRice 15y ago

Absolutely! I just wrote a reply where I alluded to that, unfortunately I didn't refresh and see this post or I would have just plugged you instead. The benefit of the Bayesian treatment here that I want to drill down on is how natural it is to adjust the prior to capture your beliefs about how items should be perceived in the presence of incomplete information. The frequentist approach is fine, but it does not provide such a pleasant, intuitive knob to tune.

jules 15y ago
Well, it depends on how you look at it. The Bayesian MAP approach is basically the same as the frequentist approach with made up data. Instead of using the maximum likelihood estimator p = pos/(pos+neg) you pretend that each new post already has some up/down votes by default, and then you use simply p = pos/(pos+neg). Seems like an even more intuitive explanation of the Bayesian knob to me! Rather than an abstract "alpha" to a "Dirichlet prior", you get something concrete (the number of made up votes). And you get a simple formula, which some would find desirable. But I agree that the Bayesian approach is conceptually much cleaner. IMO the frequentist approach is just computational corner cutting for when the math in the Bayesian approach gets too involved, which is sometimes useful. What's nice about the Bayesian approach is that you state your assumptions and then it's just turning the math machinery. In contrast, in the frequentist approach the assumptions are interwoven and hidden in arbitrary choices in how the math is done (And then they claim that Bayesians are subjective! It's just that Bayesians admit that they are subjective. Frequentists try to hide the fact that they are more subjective in the math). The not so nice thing is that turning the math machinery is not always so easy and does not always produce fast algorithms. That's where maximum likelihood and friends come in, but I'd view them as an approximation to Bayesian methods.
- NathanRice 15y ago
  Yes, the explanation of "imaginary votes" is by far the simplest way of thinking about the situation. Having mathematical formalism and a rigorously studied methods makes me feel OK about doing it though, otherwise I would be very uncomfortable with the approach :) I love Bayesian statistics from a conceptual point of view, but the ease with which one ventures into the land of analytic intractability kind of puts me off more complex models. MCMC is such a clumsy tool (in addition to taking forever); variational methods look interesting to me but I don't really feel they are quite there yet.

Eliezer 15y ago

I read the article, went "The ^&#$?" and came here to post this, more or less. It's amazing the $&#! people will get up to when they don't know Bayesian statistics.

moultano 15y ago
That really isn't fair to what this is doing. If a bayesian wanted to lower bound the distribution, they would be using exactly the same formula, but with an explicit prior. The typical bayesian solution to this of the dirichlet prior has a significantly different convergence rate, 1/n vs 1/sqrt(n) so the resulting ordering is substantially different as well, and I would argue worse for this application. The method of setting a pessimistic prior cares much less about the number of votes for a popular item than does the method of lower bounding the distribution.
- Eliezer 15y ago
  If you think most comments are bad, and you have an explicit prior that reflects this, then your ranking will update to "Hey, the comment was actually good!" at the optimal speed - no more, no less - given the rate of incoming ratings. Why would any other rate of updating be better?
  - moultano 15y ago
    Because there's a high variance on your estimate, so in a lot of circumstances just by variance you'll show users bad comments on top most of the time, rather than showing them the good comment that you are certain of. In this context, the pessimistic prior is more of a hack than the confidence interval, because your mental model isn't that most comments suck. Your mental model is that most of them are ok. The pessimistic prior is just there to say, if 1000 people confirm that a comment is ok, we should show that over a comment for which we don't know whether it's ok or not yet, and this reasoning is much better modeled by a lower bound.