8 ms·
Deeply Moving: Deep Learning for Sentiment Analysis
- nfoz 13y ago"In constrast, our new deep learning model actually builds up a representation of whole sentences based on the sentence structure. It computes the sentiment based on how words compose the meaning of longer phrases. This way, the model is not as easily fooled as previous models." Wait, that is all that's necessary to invoke the term "deep learning"?? Wow. Wikipedia seems to agree. I had thought there was something more to this buzzword than "slightly less shallow heuristic guesswork than the AI you already know and love".
- dbecker 13y agoTo possibly clarify, the way sentences are structured is itself learned, rather than relying on hard-coded rules. Maybe you still find deep-learning disappointing, but there have been some successes from having the computer learn multiple levels of internal structure/representation from data.
- anaphor 13y agoThe problem is that many sentences (at least in English) exhibit some structural ambiguity (more than one valid syntax tree). Also you have to take into account pragmatics and sociolinguistic factors (like variation), and so on. You can't just feed a formal grammar to a program and expect it to correctly parse English. We're still a LONG way from being able to correctly parse all (or even 99%) of possible English sentences.
- sp332 13y agoIt does not depend on hard-coded rules. It learns various probabilities for sentence structures. The result is much more flexible than a formal grammar.
- anaphor 13y agoTrue, but that still means you require some decent priors about the text that someone reading it might have. People generally have an idea of what the person who wrote it might be like, what sort of things to disregard, and so on. My point was that learning the "structure" of the language is not enough. I don't know exactly how this program works though, so it might be decent at that, I haven't read up on it enough.
- sp332 13y agoThe program doesn't need to know priors, if you make sure the priors for the test set are close to the same as the ones for the training set :)
- Wilya 13y agoDeep learning is just a technique to do learning in (possibly many) layers. And I disagree that trying to actually understand sentences is merely "slightly less shallow" than existing sentiment analysis systems (which, for the most part, treat sentences a just a bunch of words, count the positive and negative words, and take the average).
- deleted 13y ago[deleted]
- dave_sullivan 13y agoIf you're interested in using deep learning for nlp, I highly suggest putting a pot of coffee on and watching this a couple times: http://www.socher.org/index.php/DeepLearningTutorial/DeepLearningTutorial http://www.socher.org/index.php/DeepLearningTutorial/DeepLea... The really neat part is moving away from bag of words representations. Bag of words was always a bit of a hack. But seriously, watch those videos, it will give you a good idea on how it all works, the op is using similar methods. Edit: thanks gallamine, I fixed the link
- gallamine 13y agoDave, you're missing the ending "l" on the URL.
- RogerL 13y agoThank you so much for this, I'm just getting going in deep learning. I've been viewing and reading Hinton's stuff, and found it tough to get traction (I am just starting, that is not intended to be a comment on Hinton, but on me).
- Kronopath 13y agoFYI, since it's not exactly obvious: in the live demo (http://nlp.stanford.edu:8080/sentiment/rntnDemo.html http://nlp.stanford.edu:8080/sentiment/rntnDemo.html), you can double-click on the tree image that it generates to see a zoomed-in version with more information, and correct improperly labeled nodes to help the algorithm learn. Try it with some Rotten Tomatoes review snippets - from the ones I tried, it got almost all of them right.
- ninjin 13y agoAlso, they have a demo that supplies you with a sentence that you can check and correct in order to teach the model [1]. It was actually easy and nice enough to keep me there for quite some time giving me a few nods/laughs and forcing me to think about the sentiment structure of the sentences. The only request I would have for a feature is a "skip" button, since at times I am not entirely sure in regards to the sentiment of a sentence (you can still reload the page though and it will have the same effect). [1]: http://nlp.stanford.edu:8080/sentiment/labeling.html http://nlp.stanford.edu:8080/sentiment/labeling.html
- joeblau 13y agoFascinating project, I worked for a data analytics company and we always had a basic test that we'd try to throw at any Sentiment analysis engine. // Negative > This is shit. // Positive > This is the shit. Most engines can't sort that out. I'm definitely going to take a more in-depth look at this.
- kurige 13y agoIt doesn't look like this one was able to sort those two out either. It gave me strongly neutral signals for both sentences, but that may be because 'shit' isn't currently tagged.
- pygy_ 13y ago"This is the shit" is a very specific idiom. Unless the engine has been specifically trained to understand it, it is not surprising that it doesn't know what to do with it.
- byoung2 13y ago"This is shit" isn't inherently negative. Consider the question "what is the brown substance in the toilet?" and you can answer a neutral "this is shit".
- gfodor 13y agoPresumably if you throw a realistic corpus of written text you will have much more occurrences of the idiom.
- Cyranix 13y agoSentiment analysis implies attitudes, which imply context. A robust sentiment analysis system must either be trained on a targeted corpus (in which entities/concepts and attitudes are well aligned) or include some way of establishing a more targeted context after having been trained on a general-purpose corpus. That is to say, people generally aren't describing the physical material occupying a toilet in a corpus of movie reviews.
- 13y ago
- tylerneylon 13y agoStructured sentiment analysis has been around for a while - meaning models that take into account the overall document as more than just a bag of words (reference below to a 2007 Google paper). There are a surprising number of challenges for those new to the problem, e.g. sarcasm or idioms. It's kind of fun to work on. http://acl.ldc.upenn.edu/P/P07/P07-1055.pdf http://acl.ldc.upenn.edu/P/P07/P07-1055.pdf
- agibsonccc 13y agoI know there's deep hate for java around here, but here's another relevant link. (You could probably shoehorn in scala or clojure as well) This is a direct tutorial for doing deep learning. http://nlp.stanford.edu/courses/NAACL2013/ http://nlp.stanford.edu/courses/NAACL2013/ Take a look at a recent homework assignment they had to do with Named Entity Recognition and Deep Learning: http://nlp.stanford.edu/~socherr/pa4_ner.pdf http://nlp.stanford.edu/~socherr/pa4_ner.pdf It's good to see neural networks being leveraged as they are. Conditional Random Fields have been dominant for sequence based classifiers for a while. We've been needing to push the envelope a bit further.
- jmcintyre 13y agoWasn't there a company that did a similar sentiment analysis on Twitter and wanted to trade the stock exchanges on that information? I wonder what happened to that.
- dkersten 13y agoThis is pretty cool. I love how in some sentences, the actual words or phrases are all marked positive, yet it is able to accurately mark the entire sentence gets correctly marked as negative. For example, from the paypal/mailpile article from the front page: Are the risks larger because we are successful? "larger because we are" is marked as positive, "successful" is marked as very positive, the rest are marked as neutral. But the entire sentence is marked as negative (which to me it certainly reads negative).
- hnriot 13y agoHow is being successful negative? If anything it's neutral, being successful is clearly good, having exposure to larger risk is negative, a net neutral.
- dkersten 13y agoBeing successful is positive, but in the context of the article (paypal not releasing their money) the statement is negative.
- ranjeetbhatia 13y agoSentiment analysis has never been this good. Sentiment analysis has never been good.
- ohashi 13y agoIt doesn't seem to understand valence shifters. It seems to have been trained on not good, but not bad is counted as negative too.
- thomasahle 13y agoIt's interesting how the decades of research in computational linguistics still come down to rating a single sentence + or -. And though they are getting better, the systems are still pretty open to 'obvious' attacks: > I have seen many good movies, this one is not one of them. > All movies, except this one, are good. > I thought this movie was going to be bad, but I was wrong. which rate positive, positive and negative. I'm full of hope that the modern neural net and linear algebra type methods will eventually crack this though!
- agibsonccc 13y agoAnother hard problem: sarcasm. it's fun to think about how to approach it.
- spenuke 13y agoYeah, so much fun.
- agibsonccc 13y agoClever. hehe That aside, has anyone saw any recent works on detecting sarcasm? There's not much advancement in this that I've seen.
- brianshaler 13y agoThis deep learning method is an example from going from bag-of-words to document level analysis. To detect sarcasm, you'd have to make another leap from document analysis to personality modeling. Humans can detect sarcasm because they^H^H^H^Hwe can build a mental model of what is expected to be said—by a particular author or about a specific topic. If a statement is sufficiently contradictory to our mental model and we are sufficiently confident of the predicted sentiment, we know it's sarcasm. Without knowing who said something or in what context it was said, even humans would fail to accurately detect sarcasm. Here are some hasty examples that could vary, depending on the author or situation: "Michael Bay really outdid himself on that one." "That Kanye West song is so well-written." "It'd fun to build that PHP."
- bhelx 13y agoDoes anyone familiar with NLP know how the sentence trees are parsed? Can something like python's nltk do that? I always assumed that was very difficult to do well.
- miket 13y agoThe sentence trees are parsed with the Stanford NLP Parser, available here: http://nlp.stanford.edu/software/lex-parser.shtml http://nlp.stanford.edu/software/lex-parser.shtml
- bhelx 13y agoCool. Thank you!
- Cyranix 13y agoSome out-of-the-box parsing can be done with NLTK and OpenNLP. Just be mindful of the fact that the quality of the parsing is fairly dependent on whether the corpora you provide are sufficiently similar to the corpora that the models (part-of-speech, parsing, chunking, etc.) were trained on.
- ninjin 13y agoParsing is one of the most competitive areas of research in NLP and yes, you are correct, doing it well is very difficult. State-of-the-art performance on newswire data is around 95% if you score local structure. miket has already replied with a well-established lexical parser. The Stanford parser is one of the most robust out there, you could also go for the BLLIP (or Charniak or Charniak and Johnson) re-ranking parser [1] that although a bit old and fiddly (the GitHub version is very solid though) tends to perform well. However, both the Stanford and BLLIP parsers are constituency parsers (now we are going to go all linguistic here) and there is an alternate paradigm of dependency grammars [2]. I have heard a couple of times that Google is running a variant of the MaltParser [3] in-house and it is a very solid piece of work (the command-line interface requires some patience though, but there is plenty of documentation). People for example in Information Extraction (IE) tends to favour dependency grammars, probably since you get relevant modifiers closer to your point of interest in the tree. Now, we could also go into richer grammars like Combinatory Categorial Grammar (CCG) [4] and Head-driven Phrase Structure Grammar (HPSG) [5] but I think this reply is long enough by now... Personally I favour dependency parsing since there is more annotated data for it across plenty of languages. Also, the paradigm can cope with a lot of linguistic wonkiness that is very much present in language other than English (free word order and non-projective structures for example) [1]: https://github.com/dmcc/bllip-parser https://github.com/dmcc/bllip-parser [2]: http://en.wikipedia.org/wiki/Dependency_grammar http://en.wikipedia.org/wiki/Dependency_grammar [3]: http://maltparser.org/ http://maltparser.org/ [4]: http://en.wikipedia.org/wiki/Combinatory_categorial_grammar http://en.wikipedia.org/wiki/Combinatory_categorial_grammar [5]: http://en.wikipedia.org/wiki/Head-driven_phrase_structure_grammar http://en.wikipedia.org/wiki/Head-driven_phrase_structure_gr...
- contingencies 13y agoEthical considerations have always kept me from investing too much time in areas of computational linguistics that seem to be predominantly used for the purposes of rapidly commercialised and now widespread mass surveillance system development. While it is an interesting area, in fact it represents a change to society that I would personally feel happy not to have accelerated. Not sure if anyone else has an opinion on the ethics, here.
- rgonzalez 13y agoI would like to get an accuraye overall result of the sentence. For example this "I found the result of this study amusing since it was far from accurate." was classified as nothing less than neutral but I was trying to see if it would pick up the sarcasm. Is sentiment analysis not yet there?
- someben 13y agoNew Ruby module on Github to import & parse the treebank corpus: https://news.ycombinator.com/item?id=6395420 https://news.ycombinator.com/item?id=6395420