7 ms·

Hacking the Hivemind: Predicting Comment Karma on Internet Forums (2014) [pdf]

70 points by lainon 10y ago

Etheryte 10y ago
Needs a [2014] in the title.
mrlyc 10y ago
That article reminds me of The Auditors in Terry Pratchett's "The Thief of Time" who try to analyse a painting by reducing it to its component molecules. There is no mention of tailoring comments and replies to the target audience. Once I adopted the mindset of a senior high school or college student and posted accordingly, I was able to increase my karma on Reddit from 1,000 per year to 1,000 per month. I still haven't managed to crack Hacker News, though. Tough audience.
- rosalinekarr 10y ago
  > I still haven't managed to crack Hacker News, though. Tough audience. Just link a bunch of studies to anything you say.[^1] It doesn't actually matter if they're related.[^2] No one really checks the sources anyway.[^3] [^1]: https://arxiv.org/abs/1502.02876 https://arxiv.org/abs/1502.02876 [^2]: https://arxiv.org/abs/1610.02728 https://arxiv.org/abs/1610.02728 [^3]: https://arxiv.org/abs/1611.03799 https://arxiv.org/abs/1611.03799
  - tjalfi 10y ago
    This hasn't been true for me. My top 3 posts [0, 1, 2] are all fairly trite. All three of these were rated much higher than posts with extensive citations. [0] https://news.ycombinator.com/item?id=13538068 https://news.ycombinator.com/item?id=13538068 [1] https://news.ycombinator.com/item?id=13441744 https://news.ycombinator.com/item?id=13441744 [2] https://news.ycombinator.com/item?id=13594765 https://news.ycombinator.com/item?id=13594765 Edited: This post already has a higher score than https://news.ycombinator.com/item?id=14180432 https://news.ycombinator.com/item?id=14180432 which has five citations.
    - mrkgnao 10y ago
      Compare this low-effort comment: https://news.ycombinator.com/item?id=14124411 https://news.ycombinator.com/item?id=14124411 to these: https://news.ycombinator.com/item?id=14149200 https://news.ycombinator.com/item?id=14149200 https://news.ycombinator.com/item?id=14016438 https://news.ycombinator.com/item?id=14016438 Getting "in" when a post has few comments and is going to attract a lot of comment readership quickly or stay on the frontpage for a while is probably more important than pure comment quality.
      - 3131s 10y ago
        Absolutely true, and especially problematic because by their nature high-quality comments will take more time to formulate and therefore always be late to the party and get less visibility.
  - Xcelerate 10y ago
    Haha no way. My highest rated posts have all been some vapid feel-good thing that I posted shortly after an article was submitted to HN. My detailed explanations that link to studies rarely get more than 10 points (same phenomenon occurs on Quora as well actually...)
    - 3131s 10y ago
      I always wonder how much these factors affect a comment's score: * Proximity to the top of the page * Time of day posted * Number of sibling comments and the score of those comments On reddit, for example, it seems impossible to get noticed once there are already a few highly-upvoted replies to the comment you're replying to.
      - fudged71 10y ago
        I've been kicking myself that I didn't take sociology in university to study conversational dynamics on threaded internet forums. It continues to fascinate me
- dantillberg 10y ago
  > I still haven't managed to crack Hacker News, though. Tough audience. I can't tell for sure it's intended, but this sounds Machiavellian. And also following your own stated advice. Compliment the audience on being tough -- tell us what we want to hear -- in order to be rewarded with upvotes. Nice work.
- minimaxir 10y ago
  I've written many, many articles/analysis about the karma systems of Reddit/Hacker News. Karma aggregates are meaningless internet points. Neither Reddit nor Hacker News, nor their users, give major impact to how much user a user has (aside for low karma aggregate thresholds for downvoting on HN), nor is there any correlation between the amount of karma and the quality of content that user outputs.
  - derefr 10y ago
    Sure, in aggregate they're meaningless internet points. As a speaker/writer, though, you can use the score on an individual comment you've made, as a rough approximation of how well it managed to communicate its intent: if a comment you've made scores really highly, it can do well to stare at it and attempt to figure out what you did that made it work for people.
    - minimaxir 10y ago
      Saying that a 20pt comment is better than a 5pt comment because solely of its intrinsic quality is confirmation bias. There are many factors which help determine the score a comment receives, some of which are noted in the paper (time-of-day comment was made, time comment was made relative to the time original post was submitted, the score of the parent, etc.) For example, this comment made today, which is literally just a quotation, is at 13pts currently: https://news.ycombinator.com/item?id=14178760 https://news.ycombinator.com/item?id=14178760 But it was a quotation in response to a top rated comment in a top rated thread.
      - derefr 10y ago
        Oh, sure, a factor of 1-10x is noise. But if you normally write comments that earn 1-5 points, and then one comment earns 300pts, there might be something to the way you wrote it. Or might not. It might just be that you wrote the rebuttal everyone wanted to the stupid comment in a very popular thread. But you should try to figure that out, because if—after eliminating sources of error—you're still left with an anomalously-high score, then the cause was probably you. More than just with highly-positive scores, though, I also tend to analyze the type of writing I put into comments that ended up with negative scores. More often than not, it seems that my own problem is that I didn't sprinkle in enough "hedging words" (like the 'more often than not' in this sentence) and then someone came along with a single counterexample and a downvote storm ensued. I wouldn't have figured out that people had that reaction if the points weren't there.
- booleandilemma 10y ago
  Just post something about React Fiber.
- blazespin 10y ago
  Does HN even respect karma? My guess is no and is just there as a red herring(not necessarily intentionally) that keeps people from cracking the real metrics it's using.
  - bike4beer 10y ago
    Mention 'parabiosys' and see how long you remain alive on HN.
  - 3131s 10y ago
    What do you mean by respect karma? Obviously the ordering of sibling comments is determined in some way by upvotes and downvotes.
- psyc 10y ago
  Just ask everyone who offers an anecdote for citations, and tell everyone which named bias their point of view is.
- PhasmaFelis 10y ago
  > Once I adopted the mindset of a senior high school or college student and posted accordingly, I was able to increase my karma on Reddit from 1,000 per year to 1,000 per month. Why, though?
chrisallick 10y ago
This seems like a really fun project to try and reproduce!
- minimaxir 10y ago
  I wrote a blog post along similar lines around when the orignal paper was published: http://minimaxir.com/2014/10/hn-comments-about-comments/ http://minimaxir.com/2014/10/hn-comments-about-comments/ Unfortunately, since comment scores are still omitted from the API, it would be hard to modernize this analysis. (and the algorithms have definitely changed over the years)
lacampbell 10y ago
I think karma mechanisms have had a net negative effect on online discussion. I don't want to have preconceived notion on how I should feel about a post - I want to make my own decisions. There's no escaping it now though, which is sad.
- bcherny 10y ago
  Given how much content is generated on online forums, it is natural to have a mechanism to surface better/relevant content (different forums define relevance differently). Otherwise you have to wade through a sea of memes and low quality content to get to something that is worth consuming.
  - krapp 10y ago
    >Otherwise you have to wade through a sea of memes and low quality content to get to something that is worth consuming. Except now you have to deal with two definitions of quality - what you want to read and what gets voted up, and those don't always correlate.
  - lacampbell 10y ago
    Given how much content is generated on online forums, it is natural to have a mechanism to surface better/relevant content (different forums define relevance differently). My key objection is that what the majority of users think is "better/relevant" should have no influence on what I think is better and relevant. I can decide for myself, I don't need the 'hivemind'. What's wrong with just a "deleting memes" policy?
    - stil 10y ago
      bcherny is saying that karma's primary function is content curation. sure, you might like a few downvoted submissions from time to time, but for most content karma scores will be saving you time sifting through a sea of content. removing content by hand requires an active and vigilant moderation team, user votes help mitigate that burden. Also, the power to 'delete' is most likely a slippery slope...
crispytx 10y ago
"one could optimize karma by posting a long, pessimistic reply to a brand-new article." I knew it! Everybody on the internet is an asshole!

70 points by lainon 10y ago

Needs a [2014] in the title.

mrlyc 10y ago

That article reminds me of The Auditors in Terry Pratchett's "The Thief of Time" who try to analyse a painting by reducing it to its component molecules. There is no mention of tailoring comments and replies to the target audience. Once I adopted the mindset of a senior high school or college student and posted accordingly, I was able to increase my karma on Reddit from 1,000 per year to 1,000 per month. I still haven't managed to crack Hacker News, though. Tough audience.

rosalinekarr 10y ago
> I still haven't managed to crack Hacker News, though. Tough audience. Just link a bunch of studies to anything you say.[^1] It doesn't actually matter if they're related.[^2] No one really checks the sources anyway.[^3] [^1]: https://arxiv.org/abs/1502.02876 https://arxiv.org/abs/1502.02876 [^2]: https://arxiv.org/abs/1610.02728 https://arxiv.org/abs/1610.02728 [^3]: https://arxiv.org/abs/1611.03799 https://arxiv.org/abs/1611.03799
- tjalfi 10y ago
  This hasn't been true for me. My top 3 posts [0, 1, 2] are all fairly trite. All three of these were rated much higher than posts with extensive citations. [0] https://news.ycombinator.com/item?id=13538068 https://news.ycombinator.com/item?id=13538068 [1] https://news.ycombinator.com/item?id=13441744 https://news.ycombinator.com/item?id=13441744 [2] https://news.ycombinator.com/item?id=13594765 https://news.ycombinator.com/item?id=13594765 Edited: This post already has a higher score than https://news.ycombinator.com/item?id=14180432 https://news.ycombinator.com/item?id=14180432 which has five citations.
  - mrkgnao 10y ago
    Compare this low-effort comment: https://news.ycombinator.com/item?id=14124411 https://news.ycombinator.com/item?id=14124411 to these: https://news.ycombinator.com/item?id=14149200 https://news.ycombinator.com/item?id=14149200 https://news.ycombinator.com/item?id=14016438 https://news.ycombinator.com/item?id=14016438 Getting "in" when a post has few comments and is going to attract a lot of comment readership quickly or stay on the frontpage for a while is probably more important than pure comment quality.
    - 3131s 10y ago
      Absolutely true, and especially problematic because by their nature high-quality comments will take more time to formulate and therefore always be late to the party and get less visibility.
- Xcelerate 10y ago
  Haha no way. My highest rated posts have all been some vapid feel-good thing that I posted shortly after an article was submitted to HN. My detailed explanations that link to studies rarely get more than 10 points (same phenomenon occurs on Quora as well actually...)
  - 3131s 10y ago
    I always wonder how much these factors affect a comment's score: * Proximity to the top of the page * Time of day posted * Number of sibling comments and the score of those comments On reddit, for example, it seems impossible to get noticed once there are already a few highly-upvoted replies to the comment you're replying to.
    - fudged71 10y ago
      I've been kicking myself that I didn't take sociology in university to study conversational dynamics on threaded internet forums. It continues to fascinate me
dantillberg 10y ago
> I still haven't managed to crack Hacker News, though. Tough audience. I can't tell for sure it's intended, but this sounds Machiavellian. And also following your own stated advice. Compliment the audience on being tough -- tell us what we want to hear -- in order to be rewarded with upvotes. Nice work.
minimaxir 10y ago
I've written many, many articles/analysis about the karma systems of Reddit/Hacker News. Karma aggregates are meaningless internet points. Neither Reddit nor Hacker News, nor their users, give major impact to how much user a user has (aside for low karma aggregate thresholds for downvoting on HN), nor is there any correlation between the amount of karma and the quality of content that user outputs.
- derefr 10y ago
  Sure, in aggregate they're meaningless internet points. As a speaker/writer, though, you can use the score on an individual comment you've made, as a rough approximation of how well it managed to communicate its intent: if a comment you've made scores really highly, it can do well to stare at it and attempt to figure out what you did that made it work for people.
  - minimaxir 10y ago
    Saying that a 20pt comment is better than a 5pt comment because solely of its intrinsic quality is confirmation bias. There are many factors which help determine the score a comment receives, some of which are noted in the paper (time-of-day comment was made, time comment was made relative to the time original post was submitted, the score of the parent, etc.) For example, this comment made today, which is literally just a quotation, is at 13pts currently: https://news.ycombinator.com/item?id=14178760 https://news.ycombinator.com/item?id=14178760 But it was a quotation in response to a top rated comment in a top rated thread.
    - derefr 10y ago
      Oh, sure, a factor of 1-10x is noise. But if you normally write comments that earn 1-5 points, and then one comment earns 300pts, there might be something to the way you wrote it. Or might not. It might just be that you wrote the rebuttal everyone wanted to the stupid comment in a very popular thread. But you should try to figure that out, because if—after eliminating sources of error—you're still left with an anomalously-high score, then the cause was probably you. More than just with highly-positive scores, though, I also tend to analyze the type of writing I put into comments that ended up with negative scores. More often than not, it seems that my own problem is that I didn't sprinkle in enough "hedging words" (like the 'more often than not' in this sentence) and then someone came along with a single counterexample and a downvote storm ensued. I wouldn't have figured out that people had that reaction if the points weren't there.
booleandilemma 10y ago
Just post something about React Fiber.
blazespin 10y ago
Does HN even respect karma? My guess is no and is just there as a red herring(not necessarily intentionally) that keeps people from cracking the real metrics it's using.
- bike4beer 10y ago
  Mention 'parabiosys' and see how long you remain alive on HN.
- 3131s 10y ago
  What do you mean by respect karma? Obviously the ordering of sibling comments is determined in some way by upvotes and downvotes.
psyc 10y ago
Just ask everyone who offers an anecdote for citations, and tell everyone which named bias their point of view is.
PhasmaFelis 10y ago
> Once I adopted the mindset of a senior high school or college student and posted accordingly, I was able to increase my karma on Reddit from 1,000 per year to 1,000 per month. Why, though?

chrisallick 10y ago

This seems like a really fun project to try and reproduce!

minimaxir 10y ago
I wrote a blog post along similar lines around when the orignal paper was published: http://minimaxir.com/2014/10/hn-comments-about-comments/ http://minimaxir.com/2014/10/hn-comments-about-comments/ Unfortunately, since comment scores are still omitted from the API, it would be hard to modernize this analysis. (and the algorithms have definitely changed over the years)

lacampbell 10y ago

I think karma mechanisms have had a net negative effect on online discussion. I don't want to have preconceived notion on how I should feel about a post - I want to make my own decisions. There's no escaping it now though, which is sad.

bcherny 10y ago
Given how much content is generated on online forums, it is natural to have a mechanism to surface better/relevant content (different forums define relevance differently). Otherwise you have to wade through a sea of memes and low quality content to get to something that is worth consuming.
- krapp 10y ago
  >Otherwise you have to wade through a sea of memes and low quality content to get to something that is worth consuming. Except now you have to deal with two definitions of quality - what you want to read and what gets voted up, and those don't always correlate.
- lacampbell 10y ago
  Given how much content is generated on online forums, it is natural to have a mechanism to surface better/relevant content (different forums define relevance differently). My key objection is that what the majority of users think is "better/relevant" should have no influence on what I think is better and relevant. I can decide for myself, I don't need the 'hivemind'. What's wrong with just a "deleting memes" policy?
  - stil 10y ago
    bcherny is saying that karma's primary function is content curation. sure, you might like a few downvoted submissions from time to time, but for most content karma scores will be saving you time sifting through a sea of content. removing content by hand requires an active and vigilant moderation team, user votes help mitigate that burden. Also, the power to 'delete' is most likely a slippery slope...

crispytx 10y ago

"one could optimize karma by posting a long, pessimistic reply to a brand-new article." I knew it! Everybody on the internet is an asshole!