3 ms·
Aw, I was hoping for an actual corpus. A couple years ago I had wanted to find a corpus of news stories or similar things in order to analyze relationships bet
by otakucode 10y ago
Aw, I was hoping for an actual corpus. A couple years ago I had wanted to find a corpus of news stories or similar things in order to analyze relationships between entities (people/places/events) mentioned. It was then I discovered that every corpus I could find was specifically only permitted to be used for natural language analysis, and content analysis was specifically forbidden. Very disappointing. Still hoping to find a dump of a few decades worth of news or something some day to play with.
- espeed 10y agoGo here, search for "news": https://catalog.ldc.upenn.edu/search https://catalog.ldc.upenn.edu/search (it's linked to from the page). Also... http://about.reuters.com/researchandstandards/corpus/ http://about.reuters.com/researchandstandards/corpus/ https://snap.stanford.edu/data/ https://snap.stanford.edu/data/ http://www.nltk.org/nltk_data/ http://www.nltk.org/nltk_data/
- mark_l_watson 10y agoTake a look at the Google book corpus. I used it several years ago by renting the largest disk/RAM server Hetzner rents and collecting the most common 1gram, 2grams,...5grams in the book corpus. Many classification, summarization, etc. systems that rely on simple bag of words approaches can be improved by including ngrams. For a simple example, to label sentiment of text, considering 2grams lets you correctly handle 'not good', where considering one word at a time, ignoring word order, etc. is not so good.