8 ms·

Reddit implements "best" comment sorting based on score confidence intervals.

31 points by zck 17y ago

fleaflicker 17y ago
was this implemented on hn as well? i noticed a change a few weeks ago.
- antirez 17y ago
  I'm not sure but by the pattern I see by naked eye I guess it's something like this: rank = (upvote-downvote)/some_time_dependent_stuff
- deleted 17y ago
  [deleted]
zck 17y ago
For more technical information, look at the page Randall linked to: http://www.evanmiller.org/how-not-to-sort-by-average-rating.html http://www.evanmiller.org/how-not-to-sort-by-average-rating....
antirez 17y ago
I like this trivial way to compute scores for similar applications: score = upvotes*(upvotes/(upvotes+downvotes)) Edit: some example output follows. 10 up, 10 down = 5.0 50 up, 50 down = 25.0 100 up, 0 down = 100.0 70 up, 30 down = 49.0 2 up, 1 down = 1.3333 10 up, 20 down = 3.3333 This removes some bias due to the time, but does not remove from the game the idea that a lot of votes are still an hint of interest. Ah, and the math is trivial ;) Note that this is just the vanilla formula, you can simply alter the weight of the different parameters changing a bit the math. Useless to say that this is only the first step to get the actual sorting. You may not want to order by score, but by rank, where rank is something like that (if the items you are sorting must be fresh): rank = score / time^alpha alpha is the "obsolescence factor".
- modernhermit 17y ago
  this still doesn't solve the problem of "newer" comments staying at the bottom of the list and never getting the chance to gain traction no matter how good they are.
  - antirez 17y ago
    to solve this is not hard mixing the time factor inside. "hot" is using this algorithm but it sucks because in a thing like comments you want to add a "barrier", that is, for the first 24 hours time will do his work, but will affect the rating less as the whole discussion gets older.
    - modernhermit 17y ago
      won't modifying the rank by mixing in a time factor make comments with a good score move away from the top over time even if they still have a good score? while the math is trivial with your suggestion, it tends to be biased in favor of volume of upvotes (regardless of the ratio of upvote/downvote), then biased in favor of new comments (again, regardless of upvote/downvote relevance)? while this will solve sorting requirements for some applications, it's not really solving the problem that most comment voting systems have where volume of votes is rewarded over quality and/or newer comments are never positioned well enough to gain any traction over comments posted in the first hour. if you end up hacking your formulas a lot to fix these shortcomings wouldn't it be just as easy to say the problem is not trivial and then leverage the complicated but statistically sound formula suggested by the author?
      - antirez 17y ago
        yes, I think your point is correct but what I don't like the concepts behind the reddit new formula, because I really think that 100 upvotes and 30 downvotes are really different than 10 upvotes and 3 downvotes. Also time should absolutely appear in the equation in my opinion in the kind of discussion site like reddit or HN, since new comments, if as good as old comments, should be able to go up. And in this latest trick there is in my opinion the best way to avoid stagnation leading to the same top comments being voted up and up again. I'll experiment a bit about this issue, since I'm very interested in this issues. I happen to run oknotiize.virgilio.it that's the main social news site in Italy, and while we have voting on comments we currently use a chronological sorting. The votes are just used to colour the background red or green. Now I want to add sorting so I'll try to design an algorithm having the following goals: a) time matters, so good old comments tend to go down, but this only happens in the first 24/48 hours, in the long run everything is much more ordered by score. b) the score of a comment is roughly up/down but it tends to get better with more votes. c) still much better a/b ratios tend to win against other comments having a lot of comments. For instance a 100 up and 50 down will win against a 10 up and 5 down in a strong way, but will be more or less the same score as a 15 up and 5 down. I'll post here my final algorithm.
ryandvm 17y ago
You know what would be another good sorting comparator? Conversationality. A comment that sparked a lot of conversation (has a lot of children) should be a good candidate for floating towards the top as well.
- csbrooks 17y ago
  Not sure about that. This would tend to promote flame wars, and (on reddit) pun threads.
- there 17y ago
  but only for a lot of second-level children, not children of one reply. reddit's stupid pun threads usually have one reply after another as a staircase instead of many replies to one quality comment.
- icefox 17y ago
  That isn't a very good sorting measure. It simply means you were the first one to comment. You can game Hacker News currently this way. If there is already comments in a topic, don't start a new thread, instead reply to the top thread no matter what they said. You will get more replies and more votes.
- signa11 17y ago
  i always liked gnus usage of scoring individual news articles by either subject/author etc etc. having that would be really nice. just allow folks to score whatever/whoever they feel like, rather than foisting your version of 'goodness' on others. make low scoring articles fade into the background...
efaith 17y ago
It seems a strong improvement, at least in principle. Bias toward early comments has been a big problem. However what I have long thought of as the main problem remains: the people who are doing the voting. Karma does not solve this problem either, by the way; arguably it exacerbates it.
hxa7241 17y ago
This depends rather on what the purpose of scoring is. If it is to filter one's reading, there is an inherent contradiction: * the reader wants some filtering, so they can just read the good stuff * the filtering is done by the readers, which requires they read more than just the good stuff How effective can this ever be? It seems a weakness in all public-contribution-based systems (including Google search to some extent, because of page-rank . . .). (I posted this thought before somewhere, but I don't know if it wasn't thought good, or that no-one much read it.)
- diN0bot 17y ago
  furthermore, it means instead of a personal filter you get a mob filter. kind of sucks. at least reddit has categories.
johnwatson11218 17y ago
I hope this can lead to more long lasting discussions. Sometimes I find a thread late and want to add something but it seems pointless when the last comments were added more than a day ago. I think this site could be better if there was a way to keep having the threads active and maybe summarized and condensed every so often.

31 points by zck 17y ago

fleaflicker 17y ago

was this implemented on hn as well? i noticed a change a few weeks ago.

antirez 17y ago
I'm not sure but by the pattern I see by naked eye I guess it's something like this: rank = (upvote-downvote)/some_time_dependent_stuff
deleted 17y ago
[deleted]

zck 17y ago

For more technical information, look at the page Randall linked to: http://www.evanmiller.org/how-not-to-sort-by-average-rating.html http://www.evanmiller.org/how-not-to-sort-by-average-rating....

antirez 17y ago

I like this trivial way to compute scores for similar applications: score = upvotes*(upvotes/(upvotes+downvotes)) Edit: some example output follows. 10 up, 10 down = 5.0 50 up, 50 down = 25.0 100 up, 0 down = 100.0 70 up, 30 down = 49.0 2 up, 1 down = 1.3333 10 up, 20 down = 3.3333 This removes some bias due to the time, but does not remove from the game the idea that a lot of votes are still an hint of interest. Ah, and the math is trivial ;) Note that this is just the vanilla formula, you can simply alter the weight of the different parameters changing a bit the math. Useless to say that this is only the first step to get the actual sorting. You may not want to order by score, but by rank, where rank is something like that (if the items you are sorting must be fresh): rank = score / time^alpha alpha is the "obsolescence factor".

modernhermit 17y ago
this still doesn't solve the problem of "newer" comments staying at the bottom of the list and never getting the chance to gain traction no matter how good they are.
- antirez 17y ago
  to solve this is not hard mixing the time factor inside. "hot" is using this algorithm but it sucks because in a thing like comments you want to add a "barrier", that is, for the first 24 hours time will do his work, but will affect the rating less as the whole discussion gets older.
  - modernhermit 17y ago
    won't modifying the rank by mixing in a time factor make comments with a good score move away from the top over time even if they still have a good score? while the math is trivial with your suggestion, it tends to be biased in favor of volume of upvotes (regardless of the ratio of upvote/downvote), then biased in favor of new comments (again, regardless of upvote/downvote relevance)? while this will solve sorting requirements for some applications, it's not really solving the problem that most comment voting systems have where volume of votes is rewarded over quality and/or newer comments are never positioned well enough to gain any traction over comments posted in the first hour. if you end up hacking your formulas a lot to fix these shortcomings wouldn't it be just as easy to say the problem is not trivial and then leverage the complicated but statistically sound formula suggested by the author?
    - antirez 17y ago
      yes, I think your point is correct but what I don't like the concepts behind the reddit new formula, because I really think that 100 upvotes and 30 downvotes are really different than 10 upvotes and 3 downvotes. Also time should absolutely appear in the equation in my opinion in the kind of discussion site like reddit or HN, since new comments, if as good as old comments, should be able to go up. And in this latest trick there is in my opinion the best way to avoid stagnation leading to the same top comments being voted up and up again. I'll experiment a bit about this issue, since I'm very interested in this issues. I happen to run oknotiize.virgilio.it that's the main social news site in Italy, and while we have voting on comments we currently use a chronological sorting. The votes are just used to colour the background red or green. Now I want to add sorting so I'll try to design an algorithm having the following goals: a) time matters, so good old comments tend to go down, but this only happens in the first 24/48 hours, in the long run everything is much more ordered by score. b) the score of a comment is roughly up/down but it tends to get better with more votes. c) still much better a/b ratios tend to win against other comments having a lot of comments. For instance a 100 up and 50 down will win against a 10 up and 5 down in a strong way, but will be more or less the same score as a 15 up and 5 down. I'll post here my final algorithm.

ryandvm 17y ago

You know what would be another good sorting comparator? Conversationality. A comment that sparked a lot of conversation (has a lot of children) should be a good candidate for floating towards the top as well.

csbrooks 17y ago
Not sure about that. This would tend to promote flame wars, and (on reddit) pun threads.
there 17y ago
but only for a lot of second-level children, not children of one reply. reddit's stupid pun threads usually have one reply after another as a staircase instead of many replies to one quality comment.
icefox 17y ago
That isn't a very good sorting measure. It simply means you were the first one to comment. You can game Hacker News currently this way. If there is already comments in a topic, don't start a new thread, instead reply to the top thread no matter what they said. You will get more replies and more votes.
signa11 17y ago
i always liked gnus usage of scoring individual news articles by either subject/author etc etc. having that would be really nice. just allow folks to score whatever/whoever they feel like, rather than foisting your version of 'goodness' on others. make low scoring articles fade into the background...

efaith 17y ago

It seems a strong improvement, at least in principle. Bias toward early comments has been a big problem. However what I have long thought of as the main problem remains: the people who are doing the voting. Karma does not solve this problem either, by the way; arguably it exacerbates it.

hxa7241 17y ago

This depends rather on what the purpose of scoring is. If it is to filter one's reading, there is an inherent contradiction: * the reader wants some filtering, so they can just read the good stuff * the filtering is done by the readers, which requires they read more than just the good stuff How effective can this ever be? It seems a weakness in all public-contribution-based systems (including Google search to some extent, because of page-rank . . .). (I posted this thought before somewhere, but I don't know if it wasn't thought good, or that no-one much read it.)

diN0bot 17y ago
furthermore, it means instead of a personal filter you get a mob filter. kind of sucks. at least reddit has categories.

johnwatson11218 17y ago

I hope this can lead to more long lasting discussions. Sometimes I find a thread late and want to add something but it seems pointless when the last comments were added more than a day ago. I think this site could be better if there was a way to keep having the threads active and maybe summarized and condensed every so often.