4 ms·

Natural Language Corpus Data: Beautiful Data

110 points by espeed 10y ago

espeed 10y ago
What's the best source/method to get the frequency distribution to build an Arithmetic coding model [0] for English (multilingual ideally) with emoticons? [0] https://en.wikipedia.org/wiki/Arithmetic_coding#Defining_a_model https://en.wikipedia.org/wiki/Arithmetic_coding#Defining_a_m...
- web007 10y ago
  Onr of the main points of arithmetic coding is that you don't have to do a frequency count first like you would for Huffman. It's efficient enough that you can build your model piecewise as you go, and only sacrifice a few bits of coding inefficiency. You're not going to find many public datasets on SMS or other emoji-using entities regardless of language. Twitter or Instagram are your best bet, but building your model on those will only work best on those types of input.
  - ooqr 10y ago
    Lots of IRC servers host their logs online.
  - espeed 10y ago
    Just discovered Matthew Rothenberg's Emojitracker project: * http://www.emojitracker.com/ http://www.emojitracker.com/ * http://github.com/mroth/emojitrack http://github.com/mroth/emojitrack * https://medium.com/@mroth/how-i-built-emojitracker-179cfd8238ac#.hhoymvyp1 https://medium.com/@mroth/how-i-built-emojitracker-179cfd823... * https://medium.com/@mroth/how-i-kept-building-emojitracker-c31378810136 https://medium.com/@mroth/how-i-kept-building-emojitracker-c...
saycheese 10y ago
(2009) Just noting this content is from 2008-2009 and the related book was published in 2009.
sonabinu 10y ago
Beautiful data is a beautiful book!
otakucode 10y ago
Aw, I was hoping for an actual corpus. A couple years ago I had wanted to find a corpus of news stories or similar things in order to analyze relationships between entities (people/places/events) mentioned. It was then I discovered that every corpus I could find was specifically only permitted to be used for natural language analysis, and content analysis was specifically forbidden. Very disappointing. Still hoping to find a dump of a few decades worth of news or something some day to play with.
- espeed 10y ago
  Go here, search for "news": https://catalog.ldc.upenn.edu/search https://catalog.ldc.upenn.edu/search (it's linked to from the page). Also... http://about.reuters.com/researchandstandards/corpus/ http://about.reuters.com/researchandstandards/corpus/ https://snap.stanford.edu/data/ https://snap.stanford.edu/data/ http://www.nltk.org/nltk_data/ http://www.nltk.org/nltk_data/
- mark_l_watson 10y ago
  Take a look at the Google book corpus. I used it several years ago by renting the largest disk/RAM server Hetzner rents and collecting the most common 1gram, 2grams,...5grams in the book corpus. Many classification, summarization, etc. systems that rely on simple bag of words approaches can be improved by including ngrams. For a simple example, to label sentiment of text, considering 2grams lets you correctly handle 'not good', where considering one word at a time, ignoring word order, etc. is not so good.
deleted 10y ago
[deleted]
rspeer 10y ago
This is a great start, but as you get more serious about NLP, keep in mind that a Web crawl from 2006 is not the same as ground truth about language. The top 100,000 words in this data include a lot of ephemeral spam. If you're looking for slang, it'll mostly be outdated. And words like "sitemap" and "Shockwave" are not as common throughout language as the Web alone would indicate. The later Google Books Ngrams [1] data sets are cleaner, but that comes at a cost. You lose spam, but also a lot of other uses of language, when you only consider text that has been published in print. It's all in the formal register. And then you also get correlated OCR errors, as in [2], and you still get data whose collection ended in 2008. So what am I suggesting you should use, if data from the Web is bad for some things and data from books is bad for others? Well, both of them, and a lot of other things too. An analysis that depends on word frequencies should use the consensus of many sources. I've been working on compiling together many of those sources, in many languages, in the Python package wordfreq [3]. Now I'm working on an update to include excellent corpus data just released by the OPUS project, including OpenSubtitles 2016 [4]. [1] http://storage.googleapis.com/books/ngrams/books/datasetsv2.html http://storage.googleapis.com/books/ngrams/books/datasetsv2.... [2] http://drhagen.com/blog/the-missing-11th-of-the-month/ http://drhagen.com/blog/the-missing-11th-of-the-month/ [3] https://github.com/LuminosoInsight/wordfreq https://github.com/LuminosoInsight/wordfreq [4] http://opus.lingfil.uu.se/OpenSubtitles2016.php http://opus.lingfil.uu.se/OpenSubtitles2016.php
- espeed 10y ago
  Hi Rob - Thanks for posting this. I took a peek at the wordfreq project -- looks interesting. Are you decomposing to a unicode normal form, such as NFKD [0] or storing words as UTF-32 (that's what Python uses internally [1]), combining multi-character code points into the 32-bit code unit [2] represented by UTF-32? [0] https://en.wikipedia.org/wiki/Unicode_equivalence#Normalization https://en.wikipedia.org/wiki/Unicode_equivalence#Normalizat... [1] https://en.wikipedia.org/wiki/UTF-32#Use https://en.wikipedia.org/wiki/UTF-32#Use [2] https://en.wikipedia.org/wiki/Character_encoding#Terminology https://en.wikipedia.org/wiki/Character_encoding#Terminology
  - rspeer 10y ago

110 points by espeed 10y ago

What's the best source/method to get the frequency distribution to build an Arithmetic coding model [0] for English (multilingual ideally) with emoticons? [0] https://en.wikipedia.org/wiki/Arithmetic_coding#Defining_a_model https://en.wikipedia.org/wiki/Arithmetic_coding#Defining_a_m...

web007 10y ago
Onr of the main points of arithmetic coding is that you don't have to do a frequency count first like you would for Huffman. It's efficient enough that you can build your model piecewise as you go, and only sacrifice a few bits of coding inefficiency. You're not going to find many public datasets on SMS or other emoji-using entities regardless of language. Twitter or Instagram are your best bet, but building your model on those will only work best on those types of input.
- ooqr 10y ago
  Lots of IRC servers host their logs online.
- espeed 10y ago
  Just discovered Matthew Rothenberg's Emojitracker project: * http://www.emojitracker.com/ http://www.emojitracker.com/ * http://github.com/mroth/emojitrack http://github.com/mroth/emojitrack * https://medium.com/@mroth/how-i-built-emojitracker-179cfd8238ac#.hhoymvyp1 https://medium.com/@mroth/how-i-built-emojitracker-179cfd823... * https://medium.com/@mroth/how-i-kept-building-emojitracker-c31378810136 https://medium.com/@mroth/how-i-kept-building-emojitracker-c...

saycheese 10y ago

(2009) Just noting this content is from 2008-2009 and the related book was published in 2009.

sonabinu 10y ago

Beautiful data is a beautiful book!

otakucode 10y ago

Aw, I was hoping for an actual corpus. A couple years ago I had wanted to find a corpus of news stories or similar things in order to analyze relationships between entities (people/places/events) mentioned. It was then I discovered that every corpus I could find was specifically only permitted to be used for natural language analysis, and content analysis was specifically forbidden. Very disappointing. Still hoping to find a dump of a few decades worth of news or something some day to play with.

espeed 10y ago
Go here, search for "news": https://catalog.ldc.upenn.edu/search https://catalog.ldc.upenn.edu/search (it's linked to from the page). Also... http://about.reuters.com/researchandstandards/corpus/ http://about.reuters.com/researchandstandards/corpus/ https://snap.stanford.edu/data/ https://snap.stanford.edu/data/ http://www.nltk.org/nltk_data/ http://www.nltk.org/nltk_data/
mark_l_watson 10y ago
Take a look at the Google book corpus. I used it several years ago by renting the largest disk/RAM server Hetzner rents and collecting the most common 1gram, 2grams,...5grams in the book corpus. Many classification, summarization, etc. systems that rely on simple bag of words approaches can be improved by including ngrams. For a simple example, to label sentiment of text, considering 2grams lets you correctly handle 'not good', where considering one word at a time, ignoring word order, etc. is not so good.

deleted 10y ago

[deleted]

rspeer 10y ago

This is a great start, but as you get more serious about NLP, keep in mind that a Web crawl from 2006 is not the same as ground truth about language. The top 100,000 words in this data include a lot of ephemeral spam. If you're looking for slang, it'll mostly be outdated. And words like "sitemap" and "Shockwave" are not as common throughout language as the Web alone would indicate. The later Google Books Ngrams [1] data sets are cleaner, but that comes at a cost. You lose spam, but also a lot of other uses of language, when you only consider text that has been published in print. It's all in the formal register. And then you also get correlated OCR errors, as in [2], and you still get data whose collection ended in 2008. So what am I suggesting you should use, if data from the Web is bad for some things and data from books is bad for others? Well, both of them, and a lot of other things too. An analysis that depends on word frequencies should use the consensus of many sources. I've been working on compiling together many of those sources, in many languages, in the Python package wordfreq [3]. Now I'm working on an update to include excellent corpus data just released by the OPUS project, including OpenSubtitles 2016 [4]. [1] http://storage.googleapis.com/books/ngrams/books/datasetsv2.html http://storage.googleapis.com/books/ngrams/books/datasetsv2.... [2] http://drhagen.com/blog/the-missing-11th-of-the-month/ http://drhagen.com/blog/the-missing-11th-of-the-month/ [3] https://github.com/LuminosoInsight/wordfreq https://github.com/LuminosoInsight/wordfreq [4] http://opus.lingfil.uu.se/OpenSubtitles2016.php http://opus.lingfil.uu.se/OpenSubtitles2016.php

espeed 10y ago
Hi Rob - Thanks for posting this. I took a peek at the wordfreq project -- looks interesting. Are you decomposing to a unicode normal form, such as NFKD [0] or storing words as UTF-32 (that's what Python uses internally [1]), combining multi-character code points into the 32-bit code unit [2] represented by UTF-32? [0] https://en.wikipedia.org/wiki/Unicode_equivalence#Normalization https://en.wikipedia.org/wiki/Unicode_equivalence#Normalizat... [1] https://en.wikipedia.org/wiki/UTF-32#Use https://en.wikipedia.org/wiki/UTF-32#Use [2] https://en.wikipedia.org/wiki/Character_encoding#Terminology https://en.wikipedia.org/wiki/Character_encoding#Terminology
- rspeer 10y ago