4 ms·

The Quality, Popularity, and Negativity of 5.6M Hacker News Comments

82 points by sinak 12y ago

emcarey 12y ago
Fascinating dive into the data-really appreciate the sentiment analysis of comments made about women and diversity.
mgraczyk 12y ago
I would be interested to see a how the "negativity index" behaves with other emotionally charged topics. What other sorts of topics lead people to use a more emotionally charged vocabulary? What if instead of titles containing (women|female|diversity), you did the same analysis with other topics associated with toxic discussion? Maybe: (israel|palestine|gaza) or (gentrification|rent control|eviction) or even (windows|microsoft) The data seem to indicate that people use more emotionally charged words, both positive and negative, when they talk about something emotional/personal like diversity. I wonder if other topics lead to the same asymmetrical increase in negativity?
- wutbrodo 12y ago
  Seriously, I found a few of the conclusions drawn from the data here to be pretty odd. I was hoping for some interesting insight but the article is really disappointing. > "The language in the former is more neutral and about the content of the article (apparently Hacker News users really like to talk about Open Source software), while the submissions about gender and diversity trend to talk about tangent topics." Does the author _really_ find it surprising that combining all topics will lead to, on average, more "neutral" language than a topic that (ostensibly) doesn't require any expertise/knowledge of esoterica? It should be obvious that the number of people who have an opinion about diversity in tech will be higher than the number of people who feel the same way about "Show HN: Virtjs, an ES6 emulation library " or "The amazing progress of LEDs" (both from the front-page right now). Note: I'm not suggesting that the latter two aren't as _interesting_, but that contributing a comment of substance is obviously going to be more difficult and thus have less people go ahead and do so.
  - RyanZAG 12y ago
    Is it just me who finds the latter two far more interesting? It seems the biggest problem with HN becoming more popular is that it's picked up all of these politically charged topics which were very seldom before (from memory). If anybody could run the numbers on the percentage of topics featuring 'diversity' from 2014 compared to earlier that would be very interesting to me. There's no shortage of places to discuss diversity and the usual political issues, but far fewer places to discuss difficult business and tech issues and having HN taken over by these 'interesting' topics is very unfortunate. Is this the case though?
- minimaxir 12y ago
  Per your suggestion, I created a word cloud of (windows|microsoft), and compared it with a word cloud of (google|android) as a counterpoint: http://i.imgur.com/CD3rFRR.png http://i.imgur.com/CD3rFRR.png ...I did not expect these results.
nopinsight 12y ago
> The minimum average points score of any given comment was about 2 points, meaning that all comments received atleast 1 upvote on average. This trend has been increasing until 2011 when it peaked at about 4.5 points. Since then, the average has trended downward, with a particularly large drop starting at 2014. > Therefore, starting in 2014, both quantity and quality are on a downward trend. The conclusion is unwarranted, because there are other changes in the system and access points. Most significantly, points are no longer displayed and the shift to mobile devices. This could affect voter's behaviors over time. For example, people may be less inclined to vote when they do not see points (the numbers could trigger voting in some cases). Voting on mobile phones is also a pain as arrows are very small there, and a lot of mistakes could be made (clicking down arrow when intending to vote up).
- minimaxir 12y ago
  If people are less inclined to upvote, then they would be less inclined to downvote as well, offseting it slightly. Regardless, if the decline could be attributed to mobile devices, it would have happened way before 2014.
  - jimminy 12y ago
    > If people are less inclined to upvote, then they would be less inclined to downvote as well, offseting it slightly. That's not exactly true. Since more people can upvote than downvote, the threshold for downvote privileges is > 325 karma. So downvoting is reserved for those that frequently interact with the service in a positive manner. Downvoting could have had almost zero impact done to it due to the small set of people with that privilege. As quantity increased, and newer less frequent users stopped upvoting, because there was no way to discern one's value in that interaction. Downvotes could have held at a similar frequency, because of it's smaller group. The decrease in upvoting could be accountable for a large portion. Also, I rarely used a mobile device for interacting with HN, until 2-3 months ago, because the interactions are quite crappy (one of the reasons they mentioned for opening the API up.) Edit: It's actually interesting, that the upvote/downvote ability wasn't considered impactful, since you mentioned it further down in the article.
tonglil 12y ago
Just a quick FYI for the author: > "The average amount of positive words in a comment made in thread about gender and diversity is 2.48 words, a little higher than the average, and is also the most frequently occuring value. However, The average amount of positive words in a comment made in thread about gender and diversity is 2.10 words, a much higher increase." I think you mean: > "However, the average amount of negative words" Good article though, great graphics and cool insights!
touristtam 12y ago
I am slightly disappointed in the lack of mention toward posting over a 24 hour period, and by extend to toward the timezone coverage of the users. Although I have seen article pointing toward this in the past.
- minimaxir 12y ago
  I had done timezone analysis for Hacker News submissions, in February: http://minimaxir.com/img/hn-submissions.png http://minimaxir.com/img/hn-submissions.png More likely than not, comments would show similar behavior. (I didn't include it in the post because those types of charts are much harder to make)
gatehouse 12y ago
Very thorough. Two things I'd like to see are comment score distributions in the 3 months or whatever before and after scores were hidden (the purple histogram), and a heat map of article tone versus comment tone.
aston 12y ago
I made the worst comment of the month list for a comment I actually thought was good (!), and now I'm really curious about the context... Is there any way to recover the thread ids or parent comments for items on the the worst comments list?
- koopajah 12y ago
  Found it quickly on https://hn.algolia.com/ https://hn.algolia.com/ it was a thread titled "Steve Jobs - To All iPhone Customers" and here is the link: https://news.ycombinator.com/item?id=51226#up_51333 https://news.ycombinator.com/item?id=51226#up_51333 But I agree, I read some of the worst comments and would be nice to get a link to the discussion for each of these to get some contexts!
serf 12y ago
> It’s worth noting that on Hacker News, you can be downvoted for being factually wrong. Also for having a differing opinion, the incorrect number of characters in the post, misaligned chakras, irregular weather patterns, cosmic rays, etc.
probably_wrong 12y ago
Nice timing coincidence: recently I decided that I would not comment on a story if I didn't have anything positive to say about it. I found it that it's a lot easier for me to focus on one tiny flaw and point it out that actually making a contribution to the article. My number of comments has really gone downwards since then, but hopefully their quality will go up.
cheshire137 12y ago
Thanks for this. Third paragraph from the bottom, you say "However, The average amount of positive words" when you meant to say "negative." Second from bottom, you misspell "necessarily."
davesque 12y ago
I enjoyed this a lot. I do think the author is tending to make his assertions too strongly e.g. saying things like "this data shows that..." instead of "this data suggests that..." It's still a fun read and thought-provoking.

82 points by sinak 12y ago

Fascinating dive into the data-really appreciate the sentiment analysis of comments made about women and diversity.

I would be interested to see a how the "negativity index" behaves with other emotionally charged topics. What other sorts of topics lead people to use a more emotionally charged vocabulary? What if instead of titles containing (women|female|diversity), you did the same analysis with other topics associated with toxic discussion? Maybe: (israel|palestine|gaza) or (gentrification|rent control|eviction) or even (windows|microsoft) The data seem to indicate that people use more emotionally charged words, both positive and negative, when they talk about something emotional/personal like diversity. I wonder if other topics lead to the same asymmetrical increase in negativity?

wutbrodo 12y ago
Seriously, I found a few of the conclusions drawn from the data here to be pretty odd. I was hoping for some interesting insight but the article is really disappointing. > "The language in the former is more neutral and about the content of the article (apparently Hacker News users really like to talk about Open Source software), while the submissions about gender and diversity trend to talk about tangent topics." Does the author _really_ find it surprising that combining all topics will lead to, on average, more "neutral" language than a topic that (ostensibly) doesn't require any expertise/knowledge of esoterica? It should be obvious that the number of people who have an opinion about diversity in tech will be higher than the number of people who feel the same way about "Show HN: Virtjs, an ES6 emulation library " or "The amazing progress of LEDs" (both from the front-page right now). Note: I'm not suggesting that the latter two aren't as _interesting_, but that contributing a comment of substance is obviously going to be more difficult and thus have less people go ahead and do so.
- RyanZAG 12y ago
  Is it just me who finds the latter two far more interesting? It seems the biggest problem with HN becoming more popular is that it's picked up all of these politically charged topics which were very seldom before (from memory). If anybody could run the numbers on the percentage of topics featuring 'diversity' from 2014 compared to earlier that would be very interesting to me. There's no shortage of places to discuss diversity and the usual political issues, but far fewer places to discuss difficult business and tech issues and having HN taken over by these 'interesting' topics is very unfortunate. Is this the case though?
minimaxir 12y ago
Per your suggestion, I created a word cloud of (windows|microsoft), and compared it with a word cloud of (google|android) as a counterpoint: http://i.imgur.com/CD3rFRR.png http://i.imgur.com/CD3rFRR.png ...I did not expect these results.

nopinsight 12y ago

> The minimum average points score of any given comment was about 2 points, meaning that all comments received atleast 1 upvote on average. This trend has been increasing until 2011 when it peaked at about 4.5 points. Since then, the average has trended downward, with a particularly large drop starting at 2014. > Therefore, starting in 2014, both quantity and quality are on a downward trend. The conclusion is unwarranted, because there are other changes in the system and access points. Most significantly, points are no longer displayed and the shift to mobile devices. This could affect voter's behaviors over time. For example, people may be less inclined to vote when they do not see points (the numbers could trigger voting in some cases). Voting on mobile phones is also a pain as arrows are very small there, and a lot of mistakes could be made (clicking down arrow when intending to vote up).

minimaxir 12y ago
If people are less inclined to upvote, then they would be less inclined to downvote as well, offseting it slightly. Regardless, if the decline could be attributed to mobile devices, it would have happened way before 2014.
- jimminy 12y ago
  > If people are less inclined to upvote, then they would be less inclined to downvote as well, offseting it slightly. That's not exactly true. Since more people can upvote than downvote, the threshold for downvote privileges is > 325 karma. So downvoting is reserved for those that frequently interact with the service in a positive manner. Downvoting could have had almost zero impact done to it due to the small set of people with that privilege. As quantity increased, and newer less frequent users stopped upvoting, because there was no way to discern one's value in that interaction. Downvotes could have held at a similar frequency, because of it's smaller group. The decrease in upvoting could be accountable for a large portion. Also, I rarely used a mobile device for interacting with HN, until 2-3 months ago, because the interactions are quite crappy (one of the reasons they mentioned for opening the API up.) Edit: It's actually interesting, that the upvote/downvote ability wasn't considered impactful, since you mentioned it further down in the article.

tonglil 12y ago

Just a quick FYI for the author: > "The average amount of positive words in a comment made in thread about gender and diversity is 2.48 words, a little higher than the average, and is also the most frequently occuring value. However, The average amount of positive words in a comment made in thread about gender and diversity is 2.10 words, a much higher increase." I think you mean: > "However, the average amount of negative words" Good article though, great graphics and cool insights!

touristtam 12y ago

I am slightly disappointed in the lack of mention toward posting over a 24 hour period, and by extend to toward the timezone coverage of the users. Although I have seen article pointing toward this in the past.

minimaxir 12y ago
I had done timezone analysis for Hacker News submissions, in February: http://minimaxir.com/img/hn-submissions.png http://minimaxir.com/img/hn-submissions.png More likely than not, comments would show similar behavior. (I didn't include it in the post because those types of charts are much harder to make)

gatehouse 12y ago

Very thorough. Two things I'd like to see are comment score distributions in the 3 months or whatever before and after scores were hidden (the purple histogram), and a heat map of article tone versus comment tone.

aston 12y ago

I made the worst comment of the month list for a comment I actually thought was good (!), and now I'm really curious about the context... Is there any way to recover the thread ids or parent comments for items on the the worst comments list?

koopajah 12y ago
Found it quickly on https://hn.algolia.com/ https://hn.algolia.com/ it was a thread titled "Steve Jobs - To All iPhone Customers" and here is the link: https://news.ycombinator.com/item?id=51226#up_51333 https://news.ycombinator.com/item?id=51226#up_51333 But I agree, I read some of the worst comments and would be nice to get a link to the discussion for each of these to get some contexts!

serf 12y ago

> It’s worth noting that on Hacker News, you can be downvoted for being factually wrong. Also for having a differing opinion, the incorrect number of characters in the post, misaligned chakras, irregular weather patterns, cosmic rays, etc.

probably_wrong 12y ago

Nice timing coincidence: recently I decided that I would not comment on a story if I didn't have anything positive to say about it. I found it that it's a lot easier for me to focus on one tiny flaw and point it out that actually making a contribution to the article. My number of comments has really gone downwards since then, but hopefully their quality will go up.

cheshire137 12y ago

Thanks for this. Third paragraph from the bottom, you say "However, The average amount of positive words" when you meant to say "negative." Second from bottom, you misspell "necessarily."

davesque 12y ago

I enjoyed this a lot. I do think the author is tending to make his assertions too strongly e.g. saying things like "this data shows that..." instead of "this data suggests that..." It's still a fun read and thought-provoking.