3 ms·

Google has also released a full-web 5-gram corpus that contains "counts for all 1,176,470,663 five-word sequences that appear at least 40 times" Its fun to try

by fizx 17y ago

Google has also released a full-web 5-gram corpus that contains "counts for all 1,176,470,663 five-word sequences that appear at least 40 times" Its fun to try training on that. http://googleresearch.blogspot.com/2006/08/all-our-n-gram-are-belong-to-you.html http://googleresearch.blogspot.com/2006/08/all-our-n-gram-ar...

raffi 17y ago
As the comments (on the google page) illustrate, this corpus is close to useless unless you're a university researcher. Commercial use of this data costs big $$$. And if you're an independent you still have to pay $150 or so to get LDC to mail the six DVDs to you. One of the painful things about commercial NLP work is lack of good datasets without restrictions. There are some treasures like WordNet (which is a lexical db). For tagged data I found the American National Corpus but the excerpt they release without restrictions is too little data and it contains only a few styles of writing. I had to spend a lot of time gathering, fleecing, and marking up my own data to make After the Deadline.
- lsb 17y ago
  Start with Wikipedia? They've got a few billion words there, and it's a 5GB download.
  - raffi 17y ago
    That is what I ended up doing. Except this is raw data. Usually NLP researchers work from corpora that are already segmented into sentences and tagged with part-of-speech tags. I had to go through the hassle of segmenting and marking up the Wikipedia and Project Gutenberg.
    - mattrepl 17y ago
      Would you consider making it available to others? Perhaps the UCI Machine Learning Repository would accept it. If not, I'm sure I could find a way to have it hosted at my university.
      - bravura 17y ago
        archive.org will host datasets. To preprocess wikipedia, I have used the following software: http://sourceforge.net/apps/mediawiki/wikiprep/index.php?title=Main_Page http://sourceforge.net/apps/mediawiki/wikiprep/index.php?tit... To remove boilerplate from gutenberg requires painfully constructed heuristics. It would be great to have software released to do that.
  - aquarin 17y ago
    Yes, I done exactly this. it is actually 20GB uncompressed and takes 5-6h to parse. http://appraisalmind.com/articles/wikipedia-word-frequency-list/ http://appraisalmind.com/articles/wikipedia-word-frequency-l...
  - Create 17y ago
    This is why GOOG silently backed Wikipedia: to gain quality content regarding syntax and semantics. Other than give people a "good reason to surf" and hit "ads" [trackers] down the road.
- abecedarius 17y ago
  It's worse than that: they tack on $30 shipping to mail those 6 DVDs. (When last I looked, anyway.)

by fizx 17y ago

raffi 17y ago

As the comments (on the google page) illustrate, this corpus is close to useless unless you're a university researcher. Commercial use of this data costs big $$$. And if you're an independent you still have to pay $150 or so to get LDC to mail the six DVDs to you. One of the painful things about commercial NLP work is lack of good datasets without restrictions. There are some treasures like WordNet (which is a lexical db). For tagged data I found the American National Corpus but the excerpt they release without restrictions is too little data and it contains only a few styles of writing. I had to spend a lot of time gathering, fleecing, and marking up my own data to make After the Deadline.

lsb 17y ago
Start with Wikipedia? They've got a few billion words there, and it's a 5GB download.
- raffi 17y ago
  That is what I ended up doing. Except this is raw data. Usually NLP researchers work from corpora that are already segmented into sentences and tagged with part-of-speech tags. I had to go through the hassle of segmenting and marking up the Wikipedia and Project Gutenberg.
  - mattrepl 17y ago
    Would you consider making it available to others? Perhaps the UCI Machine Learning Repository would accept it. If not, I'm sure I could find a way to have it hosted at my university.
    - bravura 17y ago
      archive.org will host datasets. To preprocess wikipedia, I have used the following software: http://sourceforge.net/apps/mediawiki/wikiprep/index.php?title=Main_Page http://sourceforge.net/apps/mediawiki/wikiprep/index.php?tit... To remove boilerplate from gutenberg requires painfully constructed heuristics. It would be great to have software released to do that.
- aquarin 17y ago
  Yes, I done exactly this. it is actually 20GB uncompressed and takes 5-6h to parse. http://appraisalmind.com/articles/wikipedia-word-frequency-list/ http://appraisalmind.com/articles/wikipedia-word-frequency-l...
- Create 17y ago
  This is why GOOG silently backed Wikipedia: to gain quality content regarding syntax and semantics. Other than give people a "good reason to surf" and hit "ads" [trackers] down the road.
abecedarius 17y ago
It's worse than that: they tack on $30 shipping to mail those 6 DVDs. (When last I looked, anyway.)