3 ms·
Start with Wikipedia? They've got a few billion words there, and it's a 5GB download.
by lsb 17y ago
Start with Wikipedia? They've got a few billion words there, and it's a 5GB download.
- raffi 17y agoThat is what I ended up doing. Except this is raw data. Usually NLP researchers work from corpora that are already segmented into sentences and tagged with part-of-speech tags. I had to go through the hassle of segmenting and marking up the Wikipedia and Project Gutenberg.
- mattrepl 17y agoWould you consider making it available to others? Perhaps the UCI Machine Learning Repository would accept it. If not, I'm sure I could find a way to have it hosted at my university.
- bravura 17y agoarchive.org will host datasets. To preprocess wikipedia, I have used the following software: http://sourceforge.net/apps/mediawiki/wikiprep/index.php?title=Main_Page http://sourceforge.net/apps/mediawiki/wikiprep/index.php?tit... To remove boilerplate from gutenberg requires painfully constructed heuristics. It would be great to have software released to do that.
- aquarin 17y agoYes, I done exactly this. it is actually 20GB uncompressed and takes 5-6h to parse. http://appraisalmind.com/articles/wikipedia-word-frequency-list/ http://appraisalmind.com/articles/wikipedia-word-frequency-l...
- Create 17y agoThis is why GOOG silently backed Wikipedia: to gain quality content regarding syntax and semantics. Other than give people a "good reason to surf" and hit "ads" [trackers] down the road.