5 ms·

Twitter Natural Language Processing

207 points by jjwiseman 12y ago

jjwiseman 12y ago
Tweet NLP from CMU. We provide a tokenizer, a part-of-speech tagger, hierarchical word clusters, and a dependency parser for tweets, along with annotated corpora and web-based annotation tools.
- aroman 12y ago
  No freaking way. Literally 2 hours ago I was in professor Kosbie's office talking about a CS term project I want to do involving Twitter and NLP... were you in the room??
  - jokoon 12y ago
    i doubt that it's so much of a rare occurrence. NLP and machine learning are pretty much popular right not. that like having 2 persons with the same first name in a group of 40 people or more.
    - rbsn 12y ago
      Birthday paradox!
  - sqrt17 12y ago
    Maybe Professor Kosbie has read all those papers ... the TwitterNLP tools were published between 2011 and 2013.
dbish 12y ago
Very cool. I did some authorship attribution using a twitter corpus from MSR during grad school. The size constraint for a tweet causes users to have a different writing style then on other document types, which makes for interesting NLP problems.
vkhuc 12y ago
Great, I've been waiting for the release of TweeboParser. Just before EMNLP14 gets started.
austinhutch 12y ago
CMU does some of the most interesting work around with data from Twitter. Some of the same people responsible for this worked on predicting NFL games with Twitter data[0] [0]https://www.cs.cmu.edu/~nasmith/papers/sinha+dyer+gimpel+smith.mlsa13.pdf https://www.cs.cmu.edu/~nasmith/papers/sinha+dyer+gimpel+smi...
- vkhuc 12y ago
  Yes, Noah Smith's NLP group at CMU is awesome. Btw, Noah is moving to UWash.
navait 12y ago
Dr. O'Connor teaches my NLP class. His work is revolutionary.
agentile 12y ago
I wonder why Peen Treebank POS tags are not the default, instead of some new custom tag syntax.
- nathell 12y ago
  The default tagset that this employs is much more coarse-grained than the Penn tagset. For example, it has only two verb tags, whereas Penn has separate tags for infinitives, gerunds, past forms, etc. On the other hand, it supports the peculiar kinds of lexemes that abound in tweets, having separate tags for hashtags, @-mentions, URLs, emoticons, etc. Twitterese English is very different from standard English, and by taking the dedicated-tagset approach that embraces this fact, the authors were able to accurately attribute structure even to text that appears gibberish at first sight. To me, this is very impressive.
- brendano 12y ago
  Part of it is genuine differences between online conversational language versus standard written English, like emoticons, Twitter-specific discourse markers, and hard-to-segment compounds or clitic constructions (see the Gimpel and Owoputi papers (2011, 2013) linked on the page, and/or the annotation guidelines document too). Part of it is just that it's easier for humans to annotate the coarse-grained POS tagset, and we didn't have many resources for annotation when we did it. These things also intersect ... for example, you'd have to figure out how dialectical English verbal auxiliaries like "finna", or the second or so word in "imma", map to PTB tags. It's possible but just takes more work and thinking through the descriptive linguistics and what you want to use it for. Someday I'd like to update the whole thing for a more PTB-like POS tagset, if it can be done well. I feel like Chris Manning's whitepaper on issues in PTB POS data convinced us (well, it convinced me, at least) that it might be a good idea to focus on making high quality tag annotations. (http://nlp.stanford.edu/pubs/CICLing2011-manning-tagging.pdf http://nlp.stanford.edu/pubs/CICLing2011-manning-tagging.pdf )
- sqrt17 12y ago
  People at UWash (Ritter et al.) have Twitter tools with a tagset closer to PTB tagset (they still have special tags for @mentions and #hashtags)
hnriot 12y ago
GPL :( can't use it
- rdtsc 12y ago
  Why not? Contact the author(s) and ask if they'll give you non-GPL license.
  - vkhuc 12y ago
    The whole project is based on various libraries. In particular, the POS tagger itself uses the OWLQN optimizer from Stanford NLP (licensed under GPL). However, it's possible to remove GPL libraries out of the POS tagger as mentioned here: https://github.com/brendano/ark-tweet-nlp/blob/master/LICENSE.txt https://github.com/brendano/ark-tweet-nlp/blob/master/LICENS...
    - hnriot 12y ago
      You may want to look at Factorie (https://github.com/factorie/factorie https://github.com/factorie/factorie), that has a decent POS tagger and it's not crippled by the license. It also has dependency parsing which works reasonably well.
      - vkhuc 12y ago
        I've been looking at Factorie for a while but haven't actually done anything heavy with it. I planned to replace the optimizer in CMU's POS tagger with the one implemented in OpenNLP to make the tagger fully Apache. Unfortunately, so busy right now. Currently, I'm running the tagger on AWS, so the GPL doesn't hurt me much. BTW, besides the POS tagger, CMU's TweeboParser depends on Turbo Parser which again is licensed under GPL.
  - hnriot 12y ago
    it's not as simple as that, the whole stack is GPL, mainly due to Standford's usage of GPL for all of their NLP. GPL is a killer for any project that's not a toy or for school.
nl 12y ago
For those that didn't click though, the tweets they show for their POS tagger are great: ikr smh he asked fir yo last name so he can add u on fb lololol and :o :/ :'( >:o (: :) >.< XD -__- o.O ;D :-) @_@ :P 8D :1 >:( :D =| ") :> .... I'd love to see what Word2Vec did for that first one.
- Xeoncross 12y ago
  The demonstrates the importance of being able to parse languages other than english.
- oneeyedpigeon 12y ago
  POS is an unfortunate acronym. I genuinely thought you were being derogatory and therefore sarcastic at first! Thus, the challenge of natural-language processing; I wonder if the tagger is actually more intelligent than I am ...
  - corin_ 12y ago
    And to me I thought "point of sale" rather than "piece of shit", lots of options!
- ryanjshaw 12y ago
  I'm scared. Does this mean there's some value to parsing this kind of gibberish? If so, does the fact that I consider it gibberish mean I'm going to miss the bus on future developments because I've decided I'm too old to take people who write like this seriously?
  - ohashi 12y ago
    Until someone starts parsing it, how would we know if there is value or not?

207 points by jjwiseman 12y ago

Tweet NLP from CMU. We provide a tokenizer, a part-of-speech tagger, hierarchical word clusters, and a dependency parser for tweets, along with annotated corpora and web-based annotation tools.

aroman 12y ago
No freaking way. Literally 2 hours ago I was in professor Kosbie's office talking about a CS term project I want to do involving Twitter and NLP... were you in the room??
- jokoon 12y ago
  i doubt that it's so much of a rare occurrence. NLP and machine learning are pretty much popular right not. that like having 2 persons with the same first name in a group of 40 people or more.
  - rbsn 12y ago
    Birthday paradox!
- sqrt17 12y ago
  Maybe Professor Kosbie has read all those papers ... the TwitterNLP tools were published between 2011 and 2013.

dbish 12y ago

Very cool. I did some authorship attribution using a twitter corpus from MSR during grad school. The size constraint for a tweet causes users to have a different writing style then on other document types, which makes for interesting NLP problems.

vkhuc 12y ago

Great, I've been waiting for the release of TweeboParser. Just before EMNLP14 gets started.

austinhutch 12y ago

CMU does some of the most interesting work around with data from Twitter. Some of the same people responsible for this worked on predicting NFL games with Twitter data[0] [0]https://www.cs.cmu.edu/~nasmith/papers/sinha+dyer+gimpel+smith.mlsa13.pdf https://www.cs.cmu.edu/~nasmith/papers/sinha+dyer+gimpel+smi...

vkhuc 12y ago
Yes, Noah Smith's NLP group at CMU is awesome. Btw, Noah is moving to UWash.

navait 12y ago

Dr. O'Connor teaches my NLP class. His work is revolutionary.

agentile 12y ago

I wonder why Peen Treebank POS tags are not the default, instead of some new custom tag syntax.

nathell 12y ago
The default tagset that this employs is much more coarse-grained than the Penn tagset. For example, it has only two verb tags, whereas Penn has separate tags for infinitives, gerunds, past forms, etc. On the other hand, it supports the peculiar kinds of lexemes that abound in tweets, having separate tags for hashtags, @-mentions, URLs, emoticons, etc. Twitterese English is very different from standard English, and by taking the dedicated-tagset approach that embraces this fact, the authors were able to accurately attribute structure even to text that appears gibberish at first sight. To me, this is very impressive.
brendano 12y ago
Part of it is genuine differences between online conversational language versus standard written English, like emoticons, Twitter-specific discourse markers, and hard-to-segment compounds or clitic constructions (see the Gimpel and Owoputi papers (2011, 2013) linked on the page, and/or the annotation guidelines document too). Part of it is just that it's easier for humans to annotate the coarse-grained POS tagset, and we didn't have many resources for annotation when we did it. These things also intersect ... for example, you'd have to figure out how dialectical English verbal auxiliaries like "finna", or the second or so word in "imma", map to PTB tags. It's possible but just takes more work and thinking through the descriptive linguistics and what you want to use it for. Someday I'd like to update the whole thing for a more PTB-like POS tagset, if it can be done well. I feel like Chris Manning's whitepaper on issues in PTB POS data convinced us (well, it convinced me, at least) that it might be a good idea to focus on making high quality tag annotations. (http://nlp.stanford.edu/pubs/CICLing2011-manning-tagging.pdf http://nlp.stanford.edu/pubs/CICLing2011-manning-tagging.pdf )
sqrt17 12y ago
People at UWash (Ritter et al.) have Twitter tools with a tagset closer to PTB tagset (they still have special tags for @mentions and #hashtags)

hnriot 12y ago

GPL :( can't use it

rdtsc 12y ago
Why not? Contact the author(s) and ask if they'll give you non-GPL license.
- vkhuc 12y ago
  The whole project is based on various libraries. In particular, the POS tagger itself uses the OWLQN optimizer from Stanford NLP (licensed under GPL). However, it's possible to remove GPL libraries out of the POS tagger as mentioned here: https://github.com/brendano/ark-tweet-nlp/blob/master/LICENSE.txt https://github.com/brendano/ark-tweet-nlp/blob/master/LICENS...
  - hnriot 12y ago
    You may want to look at Factorie (https://github.com/factorie/factorie https://github.com/factorie/factorie), that has a decent POS tagger and it's not crippled by the license. It also has dependency parsing which works reasonably well.
    - vkhuc 12y ago
      I've been looking at Factorie for a while but haven't actually done anything heavy with it. I planned to replace the optimizer in CMU's POS tagger with the one implemented in OpenNLP to make the tagger fully Apache. Unfortunately, so busy right now. Currently, I'm running the tagger on AWS, so the GPL doesn't hurt me much. BTW, besides the POS tagger, CMU's TweeboParser depends on Turbo Parser which again is licensed under GPL.
- hnriot 12y ago
  it's not as simple as that, the whole stack is GPL, mainly due to Standford's usage of GPL for all of their NLP. GPL is a killer for any project that's not a toy or for school.

nl 12y ago

For those that didn't click though, the tweets they show for their POS tagger are great: ikr smh he asked fir yo last name so he can add u on fb lololol and :o :/ :'( >:o (: :) >.< XD -__- o.O ;D :-) @_@ :P 8D :1 >:( :D =| ") :> .... I'd love to see what Word2Vec did for that first one.

Xeoncross 12y ago
The demonstrates the importance of being able to parse languages other than english.
oneeyedpigeon 12y ago
POS is an unfortunate acronym. I genuinely thought you were being derogatory and therefore sarcastic at first! Thus, the challenge of natural-language processing; I wonder if the tagger is actually more intelligent than I am ...
- corin_ 12y ago
  And to me I thought "point of sale" rather than "piece of shit", lots of options!
ryanjshaw 12y ago
I'm scared. Does this mean there's some value to parsing this kind of gibberish? If so, does the fact that I consider it gibberish mean I'm going to miss the bus on future developments because I've decided I'm too old to take people who write like this seriously?
- ohashi 12y ago
  Until someone starts parsing it, how would we know if there is value or not?