4 ms·

I made this over the last two days in Node.js. The analysis is still pretty simple, and I'd like to expand it over time. It updates every 30 minutes, and you ca

by ddod 14y ago

I made this over the last two days in Node.js. The analysis is still pretty simple, and I'd like to expand it over time. It updates every 30 minutes, and you can already see that there's some significant shifts in literacy between the daytime and very early morning. I had originally started the project to grab Flesch-Kincaid grade levels, but it was taking me 45 minutes to analyze about 20 pages of comments, so I switched it out for Automated Readability Index. I really wanted to include HN, as I feel it's in a period of social shift, but it wouldn't have the same sort of statistical significance due to its size (it would most likely just be meaningless jarring jagged lines across the graphs). If someone can think of a good way to include HN, I'm all ears. I'd also be interested in other methods of writing analysis that I could include.

ninetax 14y ago
This is a cool idea. Did you account for the smaller amount of users at night vs day? It might be even more interesting to do comparisons between subreddits and comparisons between hashtags or followers of certain figures. Also do you have a link to the source? Keep up the good stuff!
- ddod 14y ago
  Thanks. It compares misspellings to correct spelling counterparts (anyways:anyway), so that, in itself, should account for sample size changes in Twitter. The Reddit sample size (should) stay constant, as I'm grabbing /all/comments every 15 seconds, which from everything I've seen is about 3 times longer than it takes for it to turn over. I have been thinking about subreddits and hashtags, but I'm not sure what would be interesting to people and still have enough statistical significance to update every half-hour/hour. If I were going to release this on Reddit, I'd have probably arranged it as a comparison between all the major subreddits. As for hashtags, their impermanence makes them a difficult measure. I don't have the source up yet, but it's pretty simple. I just get the json output of /all/comments and get the Twitter streams for the words I'm checking with ntwitter and then do the basic math to analyze it every half hour. I render the page myself with the backlog of values and then send it updated info as I process it, so if you leave the page open, it should keep fresh.
  - ninetax 14y ago
    Maybe you could do a one time analysis of different subreddits rather than a time series analysis?
xaa 14y ago
What exactly is Twitter sample size? If it means "the number of records processed", why does it vary over time? One way to approach the "significance" problem for smaller communities like HN is to create larger bins of misspellings/correct spellings. However, I don't think you're going to see many "anyways" or "yea" on HN at all, much less significant fluctuations over time. Finally, an interesting question might be to what extent individuals fluctuate in their ARI/FK levels in different contexts. What if a poster in /r/lolcats writes a very asinine comment but then goes five minutes later to /r/programming to speak intelligently. Or is it the case that an idiot is an idiot, regardless of context?
- ddod 14y ago
  Sample size is the anyway|anyways|yeah|yea instances. I get those as they get posted, so that's why it fluctuates with time. Since they're so commonly used, it should incidentally give you an idea of all of Twitter's load. I'm also grabbing some other words that I haven't implemented on the clientside yet, but I don't include them in the sample size. HN comments tend to be more varied and sparse, so I think you're right that measuring yea:yeah, etc. wouldn't be too enlightening. That said, I've noticed a defined qualitative shift in HN comments over the past few months, and I'd like to develop ways of measuring that before they reach Reddit/Twitter levels. As for your last point, I could track individuals but a relational comparison based off of /all/ data would be pretty difficult due to the number of comments vs. the few number of any individual's comments. Also, ARI isn't a great metric (hence me putting it in a tiny graph) because it measures chars instead of syllables. For example, "FFFFFUUUUUUUUU" has the same score as "constructivism".
nollidge 14y ago
I like the general idea and presentation, but if you're seriously desiring to measure social shift, I think basing that judgment on "yea/yeah" and "anyway/anyways" usage is problematic. The former is dropping a silent letter from an arbitrarily-spelled informal word. The latter is a difference in dialect (i.e. culture), not intelligence.

by ddod 14y ago

ninetax 14y ago

This is a cool idea. Did you account for the smaller amount of users at night vs day? It might be even more interesting to do comparisons between subreddits and comparisons between hashtags or followers of certain figures. Also do you have a link to the source? Keep up the good stuff!

ddod 14y ago
Thanks. It compares misspellings to correct spelling counterparts (anyways:anyway), so that, in itself, should account for sample size changes in Twitter. The Reddit sample size (should) stay constant, as I'm grabbing /all/comments every 15 seconds, which from everything I've seen is about 3 times longer than it takes for it to turn over. I have been thinking about subreddits and hashtags, but I'm not sure what would be interesting to people and still have enough statistical significance to update every half-hour/hour. If I were going to release this on Reddit, I'd have probably arranged it as a comparison between all the major subreddits. As for hashtags, their impermanence makes them a difficult measure. I don't have the source up yet, but it's pretty simple. I just get the json output of /all/comments and get the Twitter streams for the words I'm checking with ntwitter and then do the basic math to analyze it every half hour. I render the page myself with the backlog of values and then send it updated info as I process it, so if you leave the page open, it should keep fresh.
- ninetax 14y ago
  Maybe you could do a one time analysis of different subreddits rather than a time series analysis?

xaa 14y ago

What exactly is Twitter sample size? If it means "the number of records processed", why does it vary over time? One way to approach the "significance" problem for smaller communities like HN is to create larger bins of misspellings/correct spellings. However, I don't think you're going to see many "anyways" or "yea" on HN at all, much less significant fluctuations over time. Finally, an interesting question might be to what extent individuals fluctuate in their ARI/FK levels in different contexts. What if a poster in /r/lolcats writes a very asinine comment but then goes five minutes later to /r/programming to speak intelligently. Or is it the case that an idiot is an idiot, regardless of context?

ddod 14y ago
Sample size is the anyway|anyways|yeah|yea instances. I get those as they get posted, so that's why it fluctuates with time. Since they're so commonly used, it should incidentally give you an idea of all of Twitter's load. I'm also grabbing some other words that I haven't implemented on the clientside yet, but I don't include them in the sample size. HN comments tend to be more varied and sparse, so I think you're right that measuring yea:yeah, etc. wouldn't be too enlightening. That said, I've noticed a defined qualitative shift in HN comments over the past few months, and I'd like to develop ways of measuring that before they reach Reddit/Twitter levels. As for your last point, I could track individuals but a relational comparison based off of /all/ data would be pretty difficult due to the number of comments vs. the few number of any individual's comments. Also, ARI isn't a great metric (hence me putting it in a tiny graph) because it measures chars instead of syllables. For example, "FFFFFUUUUUUUUU" has the same score as "constructivism".

nollidge 14y ago

I like the general idea and presentation, but if you're seriously desiring to measure social shift, I think basing that judgment on "yea/yeah" and "anyway/anyways" usage is problematic. The former is dropping a silent letter from an arbitrarily-spelled informal word. The latter is a difference in dialect (i.e. culture), not intelligence.