5 ms·

Ask HN: The HN karma paradox

26 points by johnnybgoode 17y ago

Am I right about any of this? It appears that a comment which makes a point that's already widely accepted, whether true or false, tends to get more upvotes than a comment which is more controversial - again, whether true or false. The controversial yet true comments are usually more interesting and valuable. (Maybe that applies even to false controversial comments.) Fortunately, I think most people here care more about saying what they want to say than about getting more upvotes. So I'm not saying something needs to be fixed or that I have a much better idea. Still, I think it's a reason to be cautious about taking someone's karma score very seriously. Forgive me if I'm just saying things you already know.

nostrademons 17y ago
I think it's more that long, detailed, well-supported points tend to get more karma than simple one-liners. IIRC, PG posted once that there was a big correlation between comment length and score. I've noticed much less of a correlation between popularity and score than I would've expected - certainly much less than at Reddit, where you can write a detailed page-long comment that completely demolishes the parent's argument, and then have it sit at -1 because voters on Reddit reflexively downvote articles that don't fit their preconceptions. Here, that same comment is often at +10 or more.
- johnnybgoode 17y ago
  I'd definitely agree that it's far worse at Reddit. And I hear you on comment length, but I took care not to mention length because it's possible for well-written, long comments to make a widely accepted point.
- eru 17y ago
  > IIRC, PG posted once that there was a big correlation between comment length and score. Yes, he wrote that. I wrote a small script to test that proposition. It does not hold up. If there is a correlation, it is only very weak. You can have the source, if you wish. The script's written in Python.
  - visitor4rmindia 17y ago
    I'm interested eru - could I see the script?
    - eru 17y ago
      I have now sanitized the program. Anyone else wants to play with it? PG, is it OK to download such massive amounts of items?
- bartl 17y ago
  On Reddit, it's the short, witty comments that get the most votes. Unfortunately. Well thought out, detailed, long comments often end up with just a few points, or worse, in the negative, because of "too long, didn't read".
- jibiki 17y ago
  If post A and post B are equally convincing and interesting, but B is twice as long, then A is clearly superior.
  - nostrademons 17y ago
    They are rarely equally convincing and interesting, though.
gojomo 17y ago
The problem is that a single upvote-downvote axis has to capture two different, sometimes orthogonal, judgments: (1) is this comment valuable or detrimental to the discussion?; and (2) do I agree or disagree with this comment? This entanglement is unavoidable as long as there's only one score per comment, given the interface and established practice, at HN and similar sites. It might work to have a second axis/score that's specifically agree/disagree. (This could take the form of left-agree and right-disagree arrows, perhaps on the right of the comment-meta-line.) Then a comment could be be wildly agreed-with without offering a karmic windfall to repetitions of obvious popular sentiments, or wildly disagreed-with without the current undercurrents of censorship (sinking/fading-out) and karmic punishment via disagreement-downvotes. Personally, I would expect to find the comments with both net upvotes (valuable) and rightvotes (disagreement) to be most interesting -- because they capture challenging minority viewpoints, but well-presented.
- johnnybgoode 17y ago
  Yes, exactly. I kind of assumed that adding a second voting axis was a nonstarter, but that would indeed provide more useful information about each comment.
- kleevr 17y ago
  edit: yeah, I'm pretty sure I agree too, I'm not sure why I wrote so much... I think strict Up-Down voting ultimately regresses to a Boo/Yay judgement; despite whatever benevolent community coercion exists to color the vote along some more complex institutional axis. (In so far as we're talking about a free public forum... substitute virtually any conception of "free" you like. e.g. if this were a pay-for community we could probably implicitly afford to have a higher expectation for the "respect" for institutional rules.) I think the concept of providing additional axis of voting could lead to very interesting community conversations. I think having a voting system that was usably different would drive adoption of the additional institutional rules (e.g. voting via the four cardinals would be interesting enough to be "forthwhile" somewhere on the web) . something like: ( ) I liked this comment. YAY! ( ) I'd like this comment to get a response. YAY for conversation! ( ) I do not think this comment needs a response. BOO for conversation! ( ) I disliked this comment. BOO! * Though I'm not sure what to make of the third one. These may not be great exemplars.
chrisbolt 17y ago
What kind of point is widely accepted and false?
- johnnybgoode 17y ago
  Rather than provide you with specific examples, which by definition many people would disagree with, I'll just ask you this: Isn't there something you believe to be true which most people don't? Edit: I can also suggest historical examples. How many things were once widely accepted to be true, but are today widely accepted to be false? Assuming today's view is correct, these things were at one time widely accepted and false.
- eru 17y ago
  For example (but that's kind of a meta-point): If you select any skill (say driving) and ask a sample of people in which quantile they place themself, you are very unlikely to get a uniform distribution. (Like: "90% of people think themself above median drivers.")
  - nostrademons 17y ago
    But they're often right. Most people don't think themselves above-median drivers, they think themselves above-average (mean) drivers, i.e. if you take all the car accidents and divide by the number of drivers, they've been involved in fewer than that number. And they'd probably be right: the median number of car accidents is significantly lower than the mean, because a small number of bad drivers account for a large number of car accidents. Not everything is uniformly distributed - in fact, most things aren't.
    - eru 17y ago
      I know that most things are strangely distributed. Even the normal distribution is far from uniform. So I was carefully phrasing my comments in terms of quantiles. Quantiles are evenly distributed by definition. People think themself "better than the typical guy" i.e. above median.
- axod 17y ago
ScottWhigham 17y ago
Are we talking "karma for a single comment/post" or "karma for a user"? The reason I ask is that I've been using HN since close to the beginning and never once have I looked at a user's karma score. User karma has no meaning to me, good or bad. Post/comment karma matters in the sorting and what appears on the front page so, to me, it's the only thing that I care about. So is your discussion more about user or post/comment karma? I suspect the latter.
- johnnybgoode 17y ago
  I guess I had both in mind. The post/comment karma for the reasons you mentioned, but also user karma - not just because people look at a user's karma score, but because it's used to determine other things. If it's used for even more purposes in the future (vote weighting, for example) its importance will only increase.
mixmax 17y ago
There are many interesting subtleties to voting and karma systems, such as the one we use here on HN. Another one is that a post that has a child that glorifies the parent "Great insight, bla.bla." will inevitably be voted up. You can probably find more if you look a bit. You assume humans are rational beings and vote accordingly. But they aren't, they are emotional beasts that are hard to control, and are full of strange biases and ideas about what is right and wrong. See for instance this list of cognitive biases from wikipedia: http://en.wikipedia.org/wiki/List_of_cognitive_biases http://en.wikipedia.org/wiki/List_of_cognitive_biases Your paradox is explained by one of the biases listed, I'll leave it to you to figure out which one ;-) It's all about psychology.
- johnnybgoode 17y ago
  When did I assume humans are rational beings and vote accordingly? What I was getting at is that widely accepted points get lots of upvotes as people express their agreement, but more controversial points get more of their upvotes because of the comment's quality. For the controversial points, this is offset by downvotes from people expressing disagreement. See gojomo's comment for a possible solution. I'm not saying any voting system can be perfect, but this particular problem is explained by the fact that multiple judgments are being captured by a single voting axis, as gojomo said.
  - mixmax 17y ago
    Reading my parent comment I see that it may look a bit arrogant, sorry that wasn't intentional, I merely think the topic is interesting, and hope to add some value to the conversation :-) I think that basically what's going on here is that a simplistic voting system, such as the one we use here, and also one that carries more parameters and dimensions will always just be a simplistic overlay on an extremely nuanced and varied set of psychological rules that humans use for evaluating their surroundings, thus making it hard to accurately have karma map onto a users insightfullness (in lack of a better word) I'm pretty sure I could get a lot of karma by using psychology tricks to make people upvote me, but it wouldn't be very interesting. Or ethical for that matter. So I absolutely agree with you: Karma isn't a measure of a users insightfullness, there's probably a correlation but it's weakened by a lot of parameters, such as the ones you describe, age of the account, and lots of other factors. And hey, Welcome to HN, hope you like it here :-)
DenisM 17y ago
Soon you will realize the pointlessness of the score system (no matter how it's implemented), and from there it's a short distance to leaving it behind and getting down to business (whatever is business to you). Treat HN as a class which you need to graduate from and it will make a lot more sense.
- johnnybgoode 17y ago
  I'm not saying I expect the score system to be perfect and "fair". That would be impossible. But it isn't pointless either, because it helps us sort the articles and comments. If it really were pointless, as you say, then even the single voting axis that exists today should be removed, right?
  - DenisM 17y ago
    > it helps us sort the articles and comments. Are you sure about that? I find it roughly as many good comments with low votes as I find with high votes. My most subjectively valuable comments remain at +2 whereas my smart-ass comments get upmodded uncomfortably high. Same with stories - half of the stories I find valuable are lingering at low scores. Voting seems to be about social status, values and group identity, not about content quality. There are people who try to vote quality but I rarely see them rising above the normal voters. I suggest you give it a try and take notes while reading both high and low rated stories/comments. Maybe you will come to different conclusions than I.
ttam 17y ago
HN and most karma-based forums follow the bandwagon effect. The exception is probably slashdot, due to it's different way of power assignment and levels of voting

26 points by johnnybgoode 17y ago

nostrademons 17y ago

I think it's more that long, detailed, well-supported points tend to get more karma than simple one-liners. IIRC, PG posted once that there was a big correlation between comment length and score. I've noticed much less of a correlation between popularity and score than I would've expected - certainly much less than at Reddit, where you can write a detailed page-long comment that completely demolishes the parent's argument, and then have it sit at -1 because voters on Reddit reflexively downvote articles that don't fit their preconceptions. Here, that same comment is often at +10 or more.

johnnybgoode 17y ago
I'd definitely agree that it's far worse at Reddit. And I hear you on comment length, but I took care not to mention length because it's possible for well-written, long comments to make a widely accepted point.
eru 17y ago
> IIRC, PG posted once that there was a big correlation between comment length and score. Yes, he wrote that. I wrote a small script to test that proposition. It does not hold up. If there is a correlation, it is only very weak. You can have the source, if you wish. The script's written in Python.
- visitor4rmindia 17y ago
  I'm interested eru - could I see the script?
  - eru 17y ago
    I have now sanitized the program. Anyone else wants to play with it? PG, is it OK to download such massive amounts of items?
bartl 17y ago
On Reddit, it's the short, witty comments that get the most votes. Unfortunately. Well thought out, detailed, long comments often end up with just a few points, or worse, in the negative, because of "too long, didn't read".
jibiki 17y ago
If post A and post B are equally convincing and interesting, but B is twice as long, then A is clearly superior.
- nostrademons 17y ago
  They are rarely equally convincing and interesting, though.

gojomo 17y ago

The problem is that a single upvote-downvote axis has to capture two different, sometimes orthogonal, judgments: (1) is this comment valuable or detrimental to the discussion?; and (2) do I agree or disagree with this comment? This entanglement is unavoidable as long as there's only one score per comment, given the interface and established practice, at HN and similar sites. It might work to have a second axis/score that's specifically agree/disagree. (This could take the form of left-agree and right-disagree arrows, perhaps on the right of the comment-meta-line.) Then a comment could be be wildly agreed-with without offering a karmic windfall to repetitions of obvious popular sentiments, or wildly disagreed-with without the current undercurrents of censorship (sinking/fading-out) and karmic punishment via disagreement-downvotes. Personally, I would expect to find the comments with both net upvotes (valuable) and rightvotes (disagreement) to be most interesting -- because they capture challenging minority viewpoints, but well-presented.

johnnybgoode 17y ago
Yes, exactly. I kind of assumed that adding a second voting axis was a nonstarter, but that would indeed provide more useful information about each comment.
kleevr 17y ago
edit: yeah, I'm pretty sure I agree too, I'm not sure why I wrote so much... I think strict Up-Down voting ultimately regresses to a Boo/Yay judgement; despite whatever benevolent community coercion exists to color the vote along some more complex institutional axis. (In so far as we're talking about a free public forum... substitute virtually any conception of "free" you like. e.g. if this were a pay-for community we could probably implicitly afford to have a higher expectation for the "respect" for institutional rules.) I think the concept of providing additional axis of voting could lead to very interesting community conversations. I think having a voting system that was usably different would drive adoption of the additional institutional rules (e.g. voting via the four cardinals would be interesting enough to be "forthwhile" somewhere on the web) . something like: ( ) I liked this comment. YAY! ( ) I'd like this comment to get a response. YAY for conversation! ( ) I do not think this comment needs a response. BOO for conversation! ( ) I disliked this comment. BOO! * Though I'm not sure what to make of the third one. These may not be great exemplars.

chrisbolt 17y ago

What kind of point is widely accepted and false?

johnnybgoode 17y ago
Rather than provide you with specific examples, which by definition many people would disagree with, I'll just ask you this: Isn't there something you believe to be true which most people don't? Edit: I can also suggest historical examples. How many things were once widely accepted to be true, but are today widely accepted to be false? Assuming today's view is correct, these things were at one time widely accepted and false.
eru 17y ago
For example (but that's kind of a meta-point): If you select any skill (say driving) and ask a sample of people in which quantile they place themself, you are very unlikely to get a uniform distribution. (Like: "90% of people think themself above median drivers.")
- nostrademons 17y ago
  But they're often right. Most people don't think themselves above-median drivers, they think themselves above-average (mean) drivers, i.e. if you take all the car accidents and divide by the number of drivers, they've been involved in fewer than that number. And they'd probably be right: the median number of car accidents is significantly lower than the mean, because a small number of bad drivers account for a large number of car accidents. Not everything is uniformly distributed - in fact, most things aren't.
  - eru 17y ago
    I know that most things are strangely distributed. Even the normal distribution is far from uniform. So I was carefully phrasing my comments in terms of quantiles. Quantiles are evenly distributed by definition. People think themself "better than the typical guy" i.e. above median.
axod 17y ago

ScottWhigham 17y ago

Are we talking "karma for a single comment/post" or "karma for a user"? The reason I ask is that I've been using HN since close to the beginning and never once have I looked at a user's karma score. User karma has no meaning to me, good or bad. Post/comment karma matters in the sorting and what appears on the front page so, to me, it's the only thing that I care about. So is your discussion more about user or post/comment karma? I suspect the latter.

johnnybgoode 17y ago
I guess I had both in mind. The post/comment karma for the reasons you mentioned, but also user karma - not just because people look at a user's karma score, but because it's used to determine other things. If it's used for even more purposes in the future (vote weighting, for example) its importance will only increase.

mixmax 17y ago

There are many interesting subtleties to voting and karma systems, such as the one we use here on HN. Another one is that a post that has a child that glorifies the parent "Great insight, bla.bla." will inevitably be voted up. You can probably find more if you look a bit. You assume humans are rational beings and vote accordingly. But they aren't, they are emotional beasts that are hard to control, and are full of strange biases and ideas about what is right and wrong. See for instance this list of cognitive biases from wikipedia: http://en.wikipedia.org/wiki/List_of_cognitive_biases http://en.wikipedia.org/wiki/List_of_cognitive_biases Your paradox is explained by one of the biases listed, I'll leave it to you to figure out which one ;-) It's all about psychology.

johnnybgoode 17y ago
When did I assume humans are rational beings and vote accordingly? What I was getting at is that widely accepted points get lots of upvotes as people express their agreement, but more controversial points get more of their upvotes because of the comment's quality. For the controversial points, this is offset by downvotes from people expressing disagreement. See gojomo's comment for a possible solution. I'm not saying any voting system can be perfect, but this particular problem is explained by the fact that multiple judgments are being captured by a single voting axis, as gojomo said.
- mixmax 17y ago
  Reading my parent comment I see that it may look a bit arrogant, sorry that wasn't intentional, I merely think the topic is interesting, and hope to add some value to the conversation :-) I think that basically what's going on here is that a simplistic voting system, such as the one we use here, and also one that carries more parameters and dimensions will always just be a simplistic overlay on an extremely nuanced and varied set of psychological rules that humans use for evaluating their surroundings, thus making it hard to accurately have karma map onto a users insightfullness (in lack of a better word) I'm pretty sure I could get a lot of karma by using psychology tricks to make people upvote me, but it wouldn't be very interesting. Or ethical for that matter. So I absolutely agree with you: Karma isn't a measure of a users insightfullness, there's probably a correlation but it's weakened by a lot of parameters, such as the ones you describe, age of the account, and lots of other factors. And hey, Welcome to HN, hope you like it here :-)

DenisM 17y ago

Soon you will realize the pointlessness of the score system (no matter how it's implemented), and from there it's a short distance to leaving it behind and getting down to business (whatever is business to you). Treat HN as a class which you need to graduate from and it will make a lot more sense.

johnnybgoode 17y ago
I'm not saying I expect the score system to be perfect and "fair". That would be impossible. But it isn't pointless either, because it helps us sort the articles and comments. If it really were pointless, as you say, then even the single voting axis that exists today should be removed, right?
- DenisM 17y ago
  > it helps us sort the articles and comments. Are you sure about that? I find it roughly as many good comments with low votes as I find with high votes. My most subjectively valuable comments remain at +2 whereas my smart-ass comments get upmodded uncomfortably high. Same with stories - half of the stories I find valuable are lingering at low scores. Voting seems to be about social status, values and group identity, not about content quality. There are people who try to vote quality but I rarely see them rising above the normal voters. I suggest you give it a try and take notes while reading both high and low rated stories/comments. Maybe you will come to different conclusions than I.

ttam 17y ago

HN and most karma-based forums follow the bandwagon effect. The exception is probably slashdot, due to it's different way of power assignment and levels of voting