5 ms·

I did something similar with a 10m Tweet dataset couple of years ago: * http://ktype.net/wiki/research:articles:progress_20110209#twitter_n-gram_results http:/

by chime 14y ago

I did something similar with a 10m Tweet dataset couple of years ago: * http://ktype.net/wiki/research:articles:progress_20110209#twitter_n-gram_results http://ktype.net/wiki/research:articles:progress_20110209#tw... * http://ktype.net/wiki/research:articles:progress_20110228?s#letter-pair_frequency_table http://ktype.net/wiki/research:articles:progress_20110228?s#... I would love to redo this analysis with newer Tweets but alas, don't know where to get a usable corpus. Any suggestions? My goal was to explore many of the concepts from norvig's http://norvig.com/spell-correct.html http://norvig.com/spell-correct.html using the Tweet dataset to build a better one-finger-keyboard and word-prediction engine for iOS.

tcwc 14y ago
Unfortunately Twitter no longer allow redistribution of tweet content (http://readwrite.com/2011/03/03/how_recent_changes_to_twitters_terms_of_service_mi http://readwrite.com/2011/03/03/how_recent_changes_to_twitte...). Datasift and Gnip offer commercial access to the full firehose, but for a sample it wouldn't take long to build your own corpus from the official Twitter streaming or search API.
ComputerGuru 14y ago
I would think tweets would be skewed, no? Slang, memes, shrtnd txt 2 avoid char lmts, http://urls/ http://urls/, amongst others?
- talaketu 14y ago
  No corpus is immune from comparison, and each will have statistical parameters that reflect it's original selection criteria. Perhaps Mayzner's corpus, apparently based on a sample from literature, exhibits a bias away from the abbreviated forms widely used in written communication today. So, if you wanted to tune your text prediction software for your phone...
  - chime 14y ago
    Precisely. I was looking for predicting informal communication patterns, not formal book/newspaper style.
lam 14y ago
@chime: I am working on something similar and it would be great to exchange ideas if you want to chat about it.

by chime 14y ago

tcwc 14y ago

Unfortunately Twitter no longer allow redistribution of tweet content (http://readwrite.com/2011/03/03/how_recent_changes_to_twitters_terms_of_service_mi http://readwrite.com/2011/03/03/how_recent_changes_to_twitte...). Datasift and Gnip offer commercial access to the full firehose, but for a sample it wouldn't take long to build your own corpus from the official Twitter streaming or search API.

ComputerGuru 14y ago

I would think tweets would be skewed, no? Slang, memes, shrtnd txt 2 avoid char lmts, http://urls/ http://urls/, amongst others?

talaketu 14y ago
No corpus is immune from comparison, and each will have statistical parameters that reflect it's original selection criteria. Perhaps Mayzner's corpus, apparently based on a sample from literature, exhibits a bias away from the abbreviated forms widely used in written communication today. So, if you wanted to tune your text prediction software for your phone...
- chime 14y ago
  Precisely. I was looking for predicting informal communication patterns, not formal book/newspaper style.

lam 14y ago

@chime: I am working on something similar and it would be great to exchange ideas if you want to chat about it.