3 ms·
What exactly is Twitter sample size? If it means "the number of records processed", why does it vary over time? One way to approach the "significance" problem
by xaa 14y ago
What exactly is Twitter sample size? If it means "the number of records processed", why does it vary over time?
One way to approach the "significance" problem for smaller communities like HN is to create larger bins of misspellings/correct spellings. However, I don't think you're going to see many "anyways" or "yea" on HN at all, much less significant fluctuations over time.
Finally, an interesting question might be to what extent individuals fluctuate in their ARI/FK levels in different contexts. What if a poster in /r/lolcats writes a very asinine comment but then goes five minutes later to /r/programming to speak intelligently. Or is it the case that an idiot is an idiot, regardless of context?
- ddod 14y agoSample size is the anyway|anyways|yeah|yea instances. I get those as they get posted, so that's why it fluctuates with time. Since they're so commonly used, it should incidentally give you an idea of all of Twitter's load. I'm also grabbing some other words that I haven't implemented on the clientside yet, but I don't include them in the sample size. HN comments tend to be more varied and sparse, so I think you're right that measuring yea:yeah, etc. wouldn't be too enlightening. That said, I've noticed a defined qualitative shift in HN comments over the past few months, and I'd like to develop ways of measuring that before they reach Reddit/Twitter levels. As for your last point, I could track individuals but a relational comparison based off of /all/ data would be pretty difficult due to the number of comments vs. the few number of any individual's comments. Also, ARI isn't a great metric (hence me putting it in a tiny graph) because it measures chars instead of syllables. For example, "FFFFFUUUUUUUUU" has the same score as "constructivism".