14 ms·

Unsupervised sentiment neuron

603 points by gdb 10y ago

YCode 10y ago
The synthetic text they generated was surprisingly realistic, despite being generic. If I were perusing a dozen reviews I probably wouldn't have spotted the AI-generated ones in the crowd.
- haddr 10y ago
  We are getting better and better with automatic text generation. I wonder who will be the copyright owner of an AI-generated text, comments, songs, etc.?
  - gallerdude 10y ago
    A weird thought: at some point AI short stories may be far more profound than our own.
    - beaconstudios 10y ago
      at the moment, AI short stories are derivative, so it's unlikely. They may well be better than the average, if trained on highly regarded works, but they're not completely novel.
      - RobertoG 10y ago
        As they say: “When you take stuff from one writer it’s plagiarism, but when you take from many writers it’s called research.”
      - haddr 10y ago
        But given the whole corpus of all human-written texts, couldn't they be as creative as us?
        jsmthrowaway 10y ago
        It depends on your philosophy on what defines creativity, and whether all creativity is derivative of observing others.
        haddr 10y ago
        I don't have one, and I guess that's still somehow unsolved problem. But there are some works on creativity and AI. This is definitely an interesting space to observe.
      - happycube 10y ago
        At the moment RNN's can't remember context, so they can make stuff that looks correct, but only on the surface. I think that'll change, eventually...
        visarga 10y ago
        We need some kind of hierarchical approach, and/or memory.
- visarga 10y ago
  That's a good observation. The code and trained model for this paper is on Github. It could be used to make a fake review generator. Dangerous.
- bakhy 10y ago
  the negative reviews are funny :D and this one sounds almost like its conscious: "I couldn’t figure out how to stop this drivel. At worst, it was going absolutely nowhere, no matter what I did.Needles to say, I skim-read the entire book. Don’t waste your time." is there a sarcasm neuron in there too?
- glangdale 10y ago
  Seriously. How long before we have these reviews all over Amazon?
  - rich_ard 10y ago
    Yesterday?
nl 10y ago
So char-by-char models is the next Word2Vec then. Pretty impressive results. It would be interesting to see how it performed for other NLP tasks. I'd be pretty interested to see how many neurons it uses to attempt something like stance detection. Data-parallelism was used across 4 Pascal Titan X gpus to speed up training and increase effective memory size. Training took approximately one month. Everytime I look at something like this I find a line like that and go: "ok that's ncie.. I'll wait for the trained model".
- rspeer 10y ago
  Yeah, part of what let word2vec make such a splash that it became the one word embedding model everyone has heard of, is that the word2vec team released their model. This is a really cool example OpenAI has, but I don't know why I should ultimately care about their character model more than anyone else's if all we've got is their description of how cool it is. I hope OpenAI defies their reputation for closedness and releases the model.
  - gdb 10y ago
    Yep weights will be up soon! EDIT: in fact, weights were up at launch: https://github.com/openai/generating-reviews-discovering-sentiment/commit/15bfb78e4d5e92d5b5129a8b6ad86b100349eb5e https://github.com/openai/generating-reviews-discovering-sen...
    - rspeer 10y ago
      Sorry for my pessimistic outlook, then! Thanks.
  - backpropaganda 10y ago
    > OpenAI defies their reputation for closedness Does not compute.
    - happycube 10y ago
      The 90's definition of Open? ;) Although in this case, they did post the weights quickly.
1024core 10y ago
I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chunk of text. Training took one month across four NVIDIA Pascal GPUs Wait, what? How did "232 examples" transform into "82 million"?? OK, I get it: they pretrained the network on the 82M reviews, and then trained the last layer to do the sentiment analysis. But you can't honestly claim that you did great with just 232 examples!
- gdb 10y ago
  Thanks for the feedback — added context to that sentence to make it more clear! The main interesting thing is that none of the Amazon data was labeled, while the 232 labeled examples were.
  - p1esk 10y ago
    Very interesting, this reminds me of the 2012 paper by Andrew Ng: Building High-level Features Using Large Scale Unsupervised Learning
  - joshuamorton 10y ago
    To further clarify: does unlabeled mean "we didn't use sentiment data" or "we were only trying to predict the next character given the prior characters", since the amazon data does come with associated 1-5 star ratings, were those used or not?
    - gdb 10y ago
      We did not use the star ratings.
      - joshuamorton 10y ago
        That's what I thought, and that makes this all the more interesting!
curuinor 10y ago
moved that needle I guess
kvh 10y ago
Impressive the abstraction NNs can achieve from just character prediction. Do the other systems they compare to also use 81M Amazon reviews for training? Seems disingenuous to claim "state-of-the-art" and "less data" if they haven't.
deleted 10y ago
[deleted]
ChuckMcM 10y ago
This is a great name for a band :-). That said, I found the paper really interesting. I tend to think about LSTM systems as series expansions and using that as an analogy don't find it unusual that you can figure out the dominant (or first) coefficient of the expansion and that it has a really strong impact on the output.
deleted 10y ago
[deleted]
gallerdude 10y ago
The neural network is savage enough to learn "I would have given it zero stars, but that was not an option." Are we humans that predictable?
- teraflop 10y ago
  The training data consisted of 82 million reviews, so I'm sure that phrase (or slight variants) occurred hundreds of thousands of times.
  - visarga 10y ago
    Could be checked by counting n-grams to see just how much it differs from other reviews.
emcq 10y ago
It's very difficult to understand what the contributions are here. From what I've read so far this feels more of a proposal for future research or a press release than advancing the state of the art. * Using large models trained on lots of data to provide the foundation for sample efficient smaller models is common. * Transfer learning, fine tuning, character RNNs is common. Were there any insights learned that give a deeper understanding of these phenomena? Not knowing too much about the sentiment space, it's hard to tell how significant the resulting model is.
- amelius 10y ago
  Also, first they write: > We were very surprised that our model learned an interpretable feature, and that simply predicting the next character in Amazon reviews resulted in discovering the concept of sentiment. And then they write: > We believe the phenomenon is not specific to our model, but is instead a general property of certain large neural networks that are trained to predict the next step or dimension in their inputs. So they can't explain why a phenomenon is occurring, but they think that it generalizes to other contexts. I find it all very unconvincing. Is this kind of writing common in the deep learning literature?
  - kleiba 10y ago
    Mind you, this is not a scientific publication but a blog post that has intentionally tried to adapt the tone to that medium, presumably to appeal to a wider audience.
    - deleted 10y ago
      [deleted]
- kleiba 10y ago
  * advancing the state of the art It says right at the top: "we get 91.8% accuracy versus the previous best of 90.2%" on a standard sentiment corpus. In addition, their method needs less training data than previous approaches. * Were there any insights learned that give a deeper understanding of these phenomena? The main appeal lies in the fact that a model trained on a (1) different and (2) very general task basically "in passing" also learned to predict sentiment (i.e., a specialized task that more or less arose from the domain the general model was trained on), and pretty much through a single neuron (out of the 4096 used). The authors speculate that this might be a general effect that could also be transferred to other prediction tasks.
sushirain 10y ago
Very interesting. I wonder if they tried to predict part-of-speech tags.
- visarga 10y ago
  That would probably work. Karpathy's character based RNN could detect semantic meaning in text and code. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/
ericjang 10y ago
Why are people being so critical about this work? Sure, the blog post provides a simplified picture about what the system is actually capable of, but it's still helpful for a non-ML audience to get a better understanding of the high-level motivation behind the work. The OpenAI folks are trying to educate the broader public as well, not just ML/AI researchers. Imagine if this discovery were made by some undergraduate student who had little experience in the traditions of how ML benchmark experiments are done, or was just starting out her ML career. Would we be just as critical? As a researcher, I like seeing shorter communications like these, as it illuminates the thinking process of the researcher. Read ML papers for the ideas, not the results :) I personally don't mind blog posts that have a bit of hyped-up publicity. It's thanks to groups like DeepMind and OpenAI that have captured public imagination on the subject and accelerated such interest in prospective students in studying ML + AI + robotics. If the hype is indeed unjustified, then it'll become irrelevant in the long-term. One caveat is that researchers should be very careful to not mislead reporters who are looking for the next "killer robots" story. But that doesn't really apply here.
- chrisfosterelli 10y ago
  I personally think they did great. They targeted the blog post at a more general audience so most people can follow and get an overhead view of the idea, and then put two giant buttons for "View code" and "Read paper" right at the top of the blog post for those who want more technical writing and working code.
  - nitrogen 10y ago
    Agreeed. As a non-ML developer, I think this is the paragraph that sells the work, even if it may be an oversimplification: We were very surprised that our model learned an interpretable feature, and that simply predicting the next character in Amazon reviews resulted in discovering the concept of sentiment. We believe the phenomenon is not specific to our model, but is instead a general property of certain large neural networks that are trained to predict the next step or dimension in their inputs I think it says something very interesting about human language and information processing in general.
- 10y ago
ericjang 10y ago
Why are people being so critical about this work? Sure, the blog post provides a simplified picture about what the system is actually capable of, but it's still helpful for a non-ML audience to get a better understanding of the high-level motivation behind the work. The OpenAI folks are trying to educate the broader public as well, not just ML/AI researchers. Imagine if this discovery were made by some undergraduate student who had little experience in the traditions of how ML benchmark experiments are done. I personally don't mind blog posts that have a bit of hyped-up publicity. It's thanks to groups like DeepMind and OpenAI that have captured public imagination on the subject and accelerated such interest in prospective students in studying ML + AI + robotics. If the hype is unjustified, then it'll become irrelevant in the long-term.
auvi 10y ago
just wondering, how many AI programs (models with complete source code) OpenAI has released?
- tshadley 10y ago
  Lot of stuff here: https://github.com/openai https://github.com/openai
aabajian 10y ago
I'm trying to understand this statement: "The sentiment neuron within our model can classify reviews as negative or positive, even though the model is trained only to predict the next character in the text." If you look closely at the colorized paragraph in their paper/website, you can see that the major sentiment jumps (e.g. from green to light-green and from light-orangish to red) occur with period characters. Perhaps the insight is that periods delineate the boundary of sentiment. For example: I like this movie. I liked this movie, but not that much. I initially hated the movie, but ended up loving it. The period tells the model that the thought has ended. My question for the team: How well does the model perform if you remove periods?
- jcoffland 10y ago
  Why would that matter? Human understanding of sentiment would also go down if you removed vital information such as punctuation.
  - aabajian 10y ago
    My point would be to see how much the model is relying on punctuation. It could provide insight as to why character-based models outperform word-based models for sentiment analysis.
- azernik 10y ago
  Note that sentiment tends to also jump at the ends of grammatical phrases. For example, "Seriously, the screenplay AND the directing were horrendous" [sudden drop in sentiment without punctuation] "and clearly done by people who could not fathom what was good about the novel." This seems to have to do with a pretty deep understanding of grammar; the model waits until it the low-level neurons have something to pass up (decoding of a complete unit of meaning) before using that to update its sentiment neuron. A lot of next-character or next-word prediction ends up working like this - internally, the model keeps some state and makes big changes to its understanding at points that have to do with the structure of the stream.
itchyjunk 10y ago
I would imagine stuff like sarcasm is still out of reach though. It seems hard for humans to understand it in text based communication. Also using anything out of the standard sentimental model might throw it off. "This product is as good as <product x> (where product x has been known to perform bad." I am just trying to think of scenarios where a sentimental model would fail. Sentimental neuron sounds fascinating too. I didn't realize individual neurons could be talked about or understood outside of the concept of the NN. I am thinking in terms of "black box" its often referenced to in some articles. Since one of the research goal for openai is to train language model on jokes[0], I wonder how this neuron would perform with a joke corpus. ---------------------------- [0] https://openai.com/requests-for-research/#funnybot https://openai.com/requests-for-research/#funnybot
- Danylon 10y ago
  http://www.parrotanalytics.com/pacific-asia-knowledge-discovery-and-data-mining-conference-2016-contest/ http://www.parrotanalytics.com/pacific-asia-knowledge-discov...
- wackspurt 10y ago
  >>>Sentimental neuron sounds fascinating too. I didn't realize individual neurons could be talked about or understood outside of the concept of the NN. I am thinking in terms of "black box" its often referenced to in some articles. Yes, I agree. I recall seeing such individual neuron analysis before in Karpathy's "The Unreasonable Effectiveness of Recurrent Neural Networks". He takes a char-rnn that was training to predict the next character for source code and finds neurons that have learned to do paranthesis/bracket opening/closing.
anonymfus 10y ago
This article is not accessible. It puts all textual examples into images and ever has some absolutely unnecessary animation. Please fix it.
- ebildsten 10y ago
  Thanks for pointing this out! We've moved the textual examples into html, added alt text for images, and will be reviewing feature posts for accessibility
changoplatanero 10y ago
What's the easiest way to make a text heatmap like the ones in their blog?
d--b 10y ago
Can someone explain what is "unsupervised" about this? I'm guessing this is what confuses me most. I think this work is interesting, although when you think about it, it's kind of normal that the model converges to a point where there is a neuron that indicates whether the review is positive or negative. There are probably a lot of other traits that can be found in the "features" layer as well. There are probably neurons that can predict the geographical location of the author, based on the words they use. There are probably neurons that can predict that the author favors short sentences over long explanations. But what makes this "unsupervised"?
- Houshalter 10y ago
  It's not labelled data. They didn't tell the model what score is associated with each review. And it learned to predict it anyway. But all it was trained to do was predict the next character.
- fiter 10y ago
  I wouldn't expect that the neurons are orthogonal on a set of features which we find interesting (sentiment, geographical location). They could be bound up in some other basis of features that we do not find interesting. Other people do not expect this because there are papers about how to incentivize neurons to correspond to interesting features.
  - wackspurt 10y ago
    >> Other people do not expect this because there are papers about how to incentivize neurons to correspond to interesting features. Could you clarify that statement? Are you saying that it was unusual for this group to find such a neuron? Also, I did not know that there are papers on how to incentivize neurons to correspond to interesting features. Could you please give me some references on those?
    - fiter 10y ago
      The paper I was thinking of is called: "InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets"[0]. I do not have experience training and investigating neural nets, but from what I read in that paper, there's no reason to presume you'll find neurons that represent a feature you're interested in. In the paper they alter the reward function to get neurons that correspond to the features they are interested in. [0] https://arxiv.org/pdf/1606.03657v1.pdf https://arxiv.org/pdf/1606.03657v1.pdf
huula 10y ago
Machine Learning has become more and more like archaeology after people start saying "empirically" more and only provide a single or limited datasets.
wackspurt 10y ago
(Apologies for the slightly incoherent post below) I've been noticing a lot of work that digs into ML model internals (as they've done here to find the sentiment neuron) to understand why they work or use them to do something. Let me recall interesting instances of this: 1. Sander Dieleman's blog post about using CNNs at Spotify to do content-based recommendations for music. He didn't write about the system performance but collected playlists that maximally activated each of the CNN filters (early layer filters picked up on primitive audio features, later ones picked up on more abstract features). The filters were essentially learning the musical elements specific to various subgenres. 2. The ELI5 - Explain Like I'm Five - Python Library. It explains the outputs of many linear classifiers. I've used it to explain why a text classifier was given a certain prediction: it highlights features to show how much or little they contribute to the prediction (dark red for negative contribution, dark green for positive contribution). 3. FairML: Auditing black-box models. Inspecting the model to find which features are important. With privacy and security concerns too! Since deep learning/machine learning is very empirical at this stage, I think improvements in instrumentation can lead to ML/DL being adopted for more kinds of problems. For example: chemical/biological data. I'd be highly curious to what new ways of inspecting such kinds of data would be insightful (we can play audio input that maximally active filters for a music-related network, we can visualize what filters are learning in an object detection network, etc.)
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
srush 10y ago
If you are interested in looking at the model in more detail, we (@harvardnlp) have uploaded the model features to LSTMVis [1]. We ran their code on amazon reviews and are showing a subset of the learned features. Haven't had a chance to look further yet, but it is interesting to play with. [1] http://lstm.seas.harvard.edu/client/pattern_finder.html?data_set=32sentiment&source=states::states&pos=110&brush=28,31&queried=true&ex_cells= http://lstm.seas.harvard.edu/client/pattern_finder.html?data...
mrfusion 10y ago
why did they do this character by character? Would word by word make sense? Other than punctuation I'm not seeing why specific characters are meaningful units.
- tshadley 10y ago
  Word by word would require adding prior knowledge of words into the system, and they're trying to "start from scratch" as much as possible.
grandalf 10y ago
This has amazing potential for use in sock puppet accounts.
tshadley 10y ago
"The selected model reaches 1.12 bits per byte." (https://arxiv.org/pdf/1704.01444.pdf https://arxiv.org/pdf/1704.01444.pdf) For context, Claude Shannon found that humans could model English text with an entropy of 0.6 to 1.3 bits per character (http://languagelog.ldc.upenn.edu/myl/Shannon1950.pdf http://languagelog.ldc.upenn.edu/myl/Shannon1950.pdf)
djangowithme 10y ago
Why is the linear combination used to train the sentiment classifier? Why does its result get taken into account? Is this linear combination between 2 different strings?
du_bing 10y ago
Train on character-by-character basis, this is really incredible, quite opposite to human's intuition about language, but it seems a brilliant idea, and OpenAI tried it out, great!
andreyk 10y ago
I think it's fair to criticize this blog post for being unclear on what exactly is novel here; pre-training is a straighforward and old idea, but the blog post does not even mention this. Having accessible write ups for AI work is great, but surely it should not be confusing to domain experts or be written in such a way as to exacerbate the rampant oversimplification or misreporting in popular press about AI. Still, it is a cool mostly-experimental/empirical result, and it's good that these blog posts exist these days. For what it's worth, the paper predictably does a better job of covering the previous work and stating what their motivation was: "The experimental and evaluation protocols may be underestimating the quality of unsupervised representation learning for sentences and documents due to certain seemingly insignificant design decisions. Hill et al. (2016) also raises concern about current evaluation tasks in their recent work which provides a thorough survey of architectures and objectives for learning unsupervised sentence representations - including the above mentioned skip-thoughts. In this work, we test whether this is the case. We focus in on the task of sentiment analysis and attempt to learn an unsupervised representation that accurately contains this concept. Mikolov et al. (2013) showed that word-level recurrent language modelling supports the learning of useful word vectors and we are interested in pushing this line of work. As an approach, we consider the popular research benchmark of byte (character) level language modelling due to its further simplicity and generality. We are also interested in evaluating this approach as it is not immediately clear whether such a low-level training objective supports the learning of high-level representations." So, they question some built in assumptions from the past by training on lower-level data (characters), with a bigger dataset and more varied evaluation. The interesting result they highlight is that a single model unit is able to perform so well with their representation: "It is an open question why our model recovers the concept of sentiment in such a precise, disentangled, interpretable, and manipulable way. It is possible that sentiment as a conditioning feature has strong predictive capability for language modelling. This is likely since sentiment is such an important component of a review" , which I tend to agree with... train a on a whole lot of reviews, it's only natural to train a regressor for review sentiment.
kamalbanga 10y ago
What they have done is semi-supervised learning (Char-RNN) + supervised training of sentiment. Another way to do is semi-supervised learning (Word2Vec) + supervised training of sentiment. If first approach works better, does it imply that character level learning is more performant than word level learning?
eanzenberg 10y ago
I think one of the most amazing parts of this is how accessible the hardware is right now. You can get world-class AI results with the cost of less than most used cars. In addition, with so many resources freely available through open-source, the ability to get started is very accessible.
stillsut 10y ago
> The model struggles the more the input text diverges from review data This is where I fear the results will fail to scale. The ability to represent 'sentiment' as one neuron, and its ground truth as uni-dimensional seems most true to corpuses of online reviews where the entire point is to communicate whether you're happy with the thing that came out of the box. Most other forms of writing communicate sentiment in a more multi-dimensional way, and the subject of sentiment is more varied than a single item shipped in a box. In otherwords, the unreasonable simplicity of modelling a complex feature like sentiment with this method, is something of an artifact of this dataset.
mdibaiee 10y ago
As far as I understand, it means that there must be a relation between a character's sentiment and what the next character can (/should) be for neural network to use this as a feature, am I right? Does this mean we have unconsciously developed a language that exposes such relations?
- deleted 10y ago
  [deleted]
- vhold 10y ago
  They muse about the reason behind the sentiment neuron in the paper. "It is an open question why our model recovers the concept of sentiment in such a precise, disentangled, interpretable, and manipulable way. It is possible that sentiment as a conditioning feature has strong predictive capability for language modelling. This is likely since sentiment is such an important component of a review." They go on to frame that as an important consideration for further work like this: "Our work highlights the sensitivity of learned representations to the data distribution they are trained on. The results make clear that it is unrealistic to expect a model trained on a corpus of books, where the two most common genres are Romance and Fantasy, to learn an encoding which preserves the exact sentiment of a review." I'm wondering if a "funniness" neuron could be discovered in a model trained on millions of jokes of various funniness, or what sorts of undiscovered meaning there is in other neurons in this model.

603 points by gdb 10y ago

YCode 10y ago

The synthetic text they generated was surprisingly realistic, despite being generic. If I were perusing a dozen reviews I probably wouldn't have spotted the AI-generated ones in the crowd.

haddr 10y ago
We are getting better and better with automatic text generation. I wonder who will be the copyright owner of an AI-generated text, comments, songs, etc.?
- gallerdude 10y ago
  A weird thought: at some point AI short stories may be far more profound than our own.
  - beaconstudios 10y ago
    at the moment, AI short stories are derivative, so it's unlikely. They may well be better than the average, if trained on highly regarded works, but they're not completely novel.
    - RobertoG 10y ago
      As they say: “When you take stuff from one writer it’s plagiarism, but when you take from many writers it’s called research.”
    - haddr 10y ago
      But given the whole corpus of all human-written texts, couldn't they be as creative as us?
      - jsmthrowaway 10y ago
        It depends on your philosophy on what defines creativity, and whether all creativity is derivative of observing others.
        haddr 10y ago
        I don't have one, and I guess that's still somehow unsolved problem. But there are some works on creativity and AI. This is definitely an interesting space to observe.
    - happycube 10y ago
      At the moment RNN's can't remember context, so they can make stuff that looks correct, but only on the surface. I think that'll change, eventually...
      - visarga 10y ago
        We need some kind of hierarchical approach, and/or memory.
visarga 10y ago
That's a good observation. The code and trained model for this paper is on Github. It could be used to make a fake review generator. Dangerous.
bakhy 10y ago
the negative reviews are funny :D and this one sounds almost like its conscious: "I couldn’t figure out how to stop this drivel. At worst, it was going absolutely nowhere, no matter what I did.Needles to say, I skim-read the entire book. Don’t waste your time." is there a sarcasm neuron in there too?
glangdale 10y ago
Seriously. How long before we have these reviews all over Amazon?
- rich_ard 10y ago
  Yesterday?

nl 10y ago

So char-by-char models is the next Word2Vec then. Pretty impressive results. It would be interesting to see how it performed for other NLP tasks. I'd be pretty interested to see how many neurons it uses to attempt something like stance detection. Data-parallelism was used across 4 Pascal Titan X gpus to speed up training and increase effective memory size. Training took approximately one month. Everytime I look at something like this I find a line like that and go: "ok that's ncie.. I'll wait for the trained model".

rspeer 10y ago
Yeah, part of what let word2vec make such a splash that it became the one word embedding model everyone has heard of, is that the word2vec team released their model. This is a really cool example OpenAI has, but I don't know why I should ultimately care about their character model more than anyone else's if all we've got is their description of how cool it is. I hope OpenAI defies their reputation for closedness and releases the model.
- gdb 10y ago
  Yep weights will be up soon! EDIT: in fact, weights were up at launch: https://github.com/openai/generating-reviews-discovering-sentiment/commit/15bfb78e4d5e92d5b5129a8b6ad86b100349eb5e https://github.com/openai/generating-reviews-discovering-sen...
  - rspeer 10y ago
    Sorry for my pessimistic outlook, then! Thanks.
- backpropaganda 10y ago
  > OpenAI defies their reputation for closedness Does not compute.
  - happycube 10y ago
    The 90's definition of Open? ;) Although in this case, they did post the weights quickly.

1024core 10y ago

I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chunk of text. Training took one month across four NVIDIA Pascal GPUs Wait, what? How did "232 examples" transform into "82 million"?? OK, I get it: they pretrained the network on the 82M reviews, and then trained the last layer to do the sentiment analysis. But you can't honestly claim that you did great with just 232 examples!

gdb 10y ago
Thanks for the feedback — added context to that sentence to make it more clear! The main interesting thing is that none of the Amazon data was labeled, while the 232 labeled examples were.
- p1esk 10y ago
  Very interesting, this reminds me of the 2012 paper by Andrew Ng: Building High-level Features Using Large Scale Unsupervised Learning
- joshuamorton 10y ago
  To further clarify: does unlabeled mean "we didn't use sentiment data" or "we were only trying to predict the next character given the prior characters", since the amazon data does come with associated 1-5 star ratings, were those used or not?
  - gdb 10y ago
    We did not use the star ratings.
    - joshuamorton 10y ago
      That's what I thought, and that makes this all the more interesting!

curuinor 10y ago

moved that needle I guess

kvh 10y ago

Impressive the abstraction NNs can achieve from just character prediction. Do the other systems they compare to also use 81M Amazon reviews for training? Seems disingenuous to claim "state-of-the-art" and "less data" if they haven't.

deleted 10y ago

[deleted]

ChuckMcM 10y ago

This is a great name for a band :-). That said, I found the paper really interesting. I tend to think about LSTM systems as series expansions and using that as an analogy don't find it unusual that you can figure out the dominant (or first) coefficient of the expansion and that it has a really strong impact on the output.

deleted 10y ago

[deleted]

gallerdude 10y ago

The neural network is savage enough to learn "I would have given it zero stars, but that was not an option." Are we humans that predictable?

teraflop 10y ago
The training data consisted of 82 million reviews, so I'm sure that phrase (or slight variants) occurred hundreds of thousands of times.
- visarga 10y ago
  Could be checked by counting n-grams to see just how much it differs from other reviews.

emcq 10y ago

It's very difficult to understand what the contributions are here. From what I've read so far this feels more of a proposal for future research or a press release than advancing the state of the art. * Using large models trained on lots of data to provide the foundation for sample efficient smaller models is common. * Transfer learning, fine tuning, character RNNs is common. Were there any insights learned that give a deeper understanding of these phenomena? Not knowing too much about the sentiment space, it's hard to tell how significant the resulting model is.

amelius 10y ago
Also, first they write: > We were very surprised that our model learned an interpretable feature, and that simply predicting the next character in Amazon reviews resulted in discovering the concept of sentiment. And then they write: > We believe the phenomenon is not specific to our model, but is instead a general property of certain large neural networks that are trained to predict the next step or dimension in their inputs. So they can't explain why a phenomenon is occurring, but they think that it generalizes to other contexts. I find it all very unconvincing. Is this kind of writing common in the deep learning literature?
- kleiba 10y ago
  Mind you, this is not a scientific publication but a blog post that has intentionally tried to adapt the tone to that medium, presumably to appeal to a wider audience.
  - deleted 10y ago
    [deleted]
kleiba 10y ago
* advancing the state of the art It says right at the top: "we get 91.8% accuracy versus the previous best of 90.2%" on a standard sentiment corpus. In addition, their method needs less training data than previous approaches. * Were there any insights learned that give a deeper understanding of these phenomena? The main appeal lies in the fact that a model trained on a (1) different and (2) very general task basically "in passing" also learned to predict sentiment (i.e., a specialized task that more or less arose from the domain the general model was trained on), and pretty much through a single neuron (out of the 4096 used). The authors speculate that this might be a general effect that could also be transferred to other prediction tasks.

sushirain 10y ago

Very interesting. I wonder if they tried to predict part-of-speech tags.

visarga 10y ago
That would probably work. Karpathy's character based RNN could detect semantic meaning in text and code. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/

ericjang 10y ago

Why are people being so critical about this work? Sure, the blog post provides a simplified picture about what the system is actually capable of, but it's still helpful for a non-ML audience to get a better understanding of the high-level motivation behind the work. The OpenAI folks are trying to educate the broader public as well, not just ML/AI researchers. Imagine if this discovery were made by some undergraduate student who had little experience in the traditions of how ML benchmark experiments are done, or was just starting out her ML career. Would we be just as critical? As a researcher, I like seeing shorter communications like these, as it illuminates the thinking process of the researcher. Read ML papers for the ideas, not the results :) I personally don't mind blog posts that have a bit of hyped-up publicity. It's thanks to groups like DeepMind and OpenAI that have captured public imagination on the subject and accelerated such interest in prospective students in studying ML + AI + robotics. If the hype is indeed unjustified, then it'll become irrelevant in the long-term. One caveat is that researchers should be very careful to not mislead reporters who are looking for the next "killer robots" story. But that doesn't really apply here.

chrisfosterelli 10y ago
I personally think they did great. They targeted the blog post at a more general audience so most people can follow and get an overhead view of the idea, and then put two giant buttons for "View code" and "Read paper" right at the top of the blog post for those who want more technical writing and working code.
- nitrogen 10y ago
  Agreeed. As a non-ML developer, I think this is the paragraph that sells the work, even if it may be an oversimplification: We were very surprised that our model learned an interpretable feature, and that simply predicting the next character in Amazon reviews resulted in discovering the concept of sentiment. We believe the phenomenon is not specific to our model, but is instead a general property of certain large neural networks that are trained to predict the next step or dimension in their inputs I think it says something very interesting about human language and information processing in general.
10y ago

ericjang 10y ago

auvi 10y ago

just wondering, how many AI programs (models with complete source code) OpenAI has released?

tshadley 10y ago
Lot of stuff here: https://github.com/openai https://github.com/openai

aabajian 10y ago

I'm trying to understand this statement: "The sentiment neuron within our model can classify reviews as negative or positive, even though the model is trained only to predict the next character in the text." If you look closely at the colorized paragraph in their paper/website, you can see that the major sentiment jumps (e.g. from green to light-green and from light-orangish to red) occur with period characters. Perhaps the insight is that periods delineate the boundary of sentiment. For example: I like this movie. I liked this movie, but not that much. I initially hated the movie, but ended up loving it. The period tells the model that the thought has ended. My question for the team: How well does the model perform if you remove periods?

jcoffland 10y ago
Why would that matter? Human understanding of sentiment would also go down if you removed vital information such as punctuation.
- aabajian 10y ago
  My point would be to see how much the model is relying on punctuation. It could provide insight as to why character-based models outperform word-based models for sentiment analysis.
azernik 10y ago
Note that sentiment tends to also jump at the ends of grammatical phrases. For example, "Seriously, the screenplay AND the directing were horrendous" [sudden drop in sentiment without punctuation] "and clearly done by people who could not fathom what was good about the novel." This seems to have to do with a pretty deep understanding of grammar; the model waits until it the low-level neurons have something to pass up (decoding of a complete unit of meaning) before using that to update its sentiment neuron. A lot of next-character or next-word prediction ends up working like this - internally, the model keeps some state and makes big changes to its understanding at points that have to do with the structure of the stream.

itchyjunk 10y ago

I would imagine stuff like sarcasm is still out of reach though. It seems hard for humans to understand it in text based communication. Also using anything out of the standard sentimental model might throw it off. "This product is as good as <product x> (where product x has been known to perform bad." I am just trying to think of scenarios where a sentimental model would fail. Sentimental neuron sounds fascinating too. I didn't realize individual neurons could be talked about or understood outside of the concept of the NN. I am thinking in terms of "black box" its often referenced to in some articles. Since one of the research goal for openai is to train language model on jokes[0], I wonder how this neuron would perform with a joke corpus. ---------------------------- [0] https://openai.com/requests-for-research/#funnybot https://openai.com/requests-for-research/#funnybot

Danylon 10y ago
http://www.parrotanalytics.com/pacific-asia-knowledge-discovery-and-data-mining-conference-2016-contest/ http://www.parrotanalytics.com/pacific-asia-knowledge-discov...
wackspurt 10y ago
>>>Sentimental neuron sounds fascinating too. I didn't realize individual neurons could be talked about or understood outside of the concept of the NN. I am thinking in terms of "black box" its often referenced to in some articles. Yes, I agree. I recall seeing such individual neuron analysis before in Karpathy's "The Unreasonable Effectiveness of Recurrent Neural Networks". He takes a char-rnn that was training to predict the next character for source code and finds neurons that have learned to do paranthesis/bracket opening/closing.

anonymfus 10y ago

This article is not accessible. It puts all textual examples into images and ever has some absolutely unnecessary animation. Please fix it.

ebildsten 10y ago
Thanks for pointing this out! We've moved the textual examples into html, added alt text for images, and will be reviewing feature posts for accessibility

changoplatanero 10y ago

What's the easiest way to make a text heatmap like the ones in their blog?

d--b 10y ago

Can someone explain what is "unsupervised" about this? I'm guessing this is what confuses me most. I think this work is interesting, although when you think about it, it's kind of normal that the model converges to a point where there is a neuron that indicates whether the review is positive or negative. There are probably a lot of other traits that can be found in the "features" layer as well. There are probably neurons that can predict the geographical location of the author, based on the words they use. There are probably neurons that can predict that the author favors short sentences over long explanations. But what makes this "unsupervised"?

Houshalter 10y ago
It's not labelled data. They didn't tell the model what score is associated with each review. And it learned to predict it anyway. But all it was trained to do was predict the next character.
fiter 10y ago
I wouldn't expect that the neurons are orthogonal on a set of features which we find interesting (sentiment, geographical location). They could be bound up in some other basis of features that we do not find interesting. Other people do not expect this because there are papers about how to incentivize neurons to correspond to interesting features.
- wackspurt 10y ago
  >> Other people do not expect this because there are papers about how to incentivize neurons to correspond to interesting features. Could you clarify that statement? Are you saying that it was unusual for this group to find such a neuron? Also, I did not know that there are papers on how to incentivize neurons to correspond to interesting features. Could you please give me some references on those?
  - fiter 10y ago
    The paper I was thinking of is called: "InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets"[0]. I do not have experience training and investigating neural nets, but from what I read in that paper, there's no reason to presume you'll find neurons that represent a feature you're interested in. In the paper they alter the reward function to get neurons that correspond to the features they are interested in. [0] https://arxiv.org/pdf/1606.03657v1.pdf https://arxiv.org/pdf/1606.03657v1.pdf

huula 10y ago

Machine Learning has become more and more like archaeology after people start saying "empirically" more and only provide a single or limited datasets.

wackspurt 10y ago

(Apologies for the slightly incoherent post below) I've been noticing a lot of work that digs into ML model internals (as they've done here to find the sentiment neuron) to understand why they work or use them to do something. Let me recall interesting instances of this: 1. Sander Dieleman's blog post about using CNNs at Spotify to do content-based recommendations for music. He didn't write about the system performance but collected playlists that maximally activated each of the CNN filters (early layer filters picked up on primitive audio features, later ones picked up on more abstract features). The filters were essentially learning the musical elements specific to various subgenres. 2. The ELI5 - Explain Like I'm Five - Python Library. It explains the outputs of many linear classifiers. I've used it to explain why a text classifier was given a certain prediction: it highlights features to show how much or little they contribute to the prediction (dark red for negative contribution, dark green for positive contribution). 3. FairML: Auditing black-box models. Inspecting the model to find which features are important. With privacy and security concerns too! Since deep learning/machine learning is very empirical at this stage, I think improvements in instrumentation can lead to ML/DL being adopted for more kinds of problems. For example: chemical/biological data. I'd be highly curious to what new ways of inspecting such kinds of data would be insightful (we can play audio input that maximally active filters for a music-related network, we can visualize what filters are learning in an object detection network, etc.)

deleted 10y ago

[deleted]

deleted 10y ago

[deleted]

srush 10y ago

If you are interested in looking at the model in more detail, we (@harvardnlp) have uploaded the model features to LSTMVis [1]. We ran their code on amazon reviews and are showing a subset of the learned features. Haven't had a chance to look further yet, but it is interesting to play with. [1] http://lstm.seas.harvard.edu/client/pattern_finder.html?data_set=32sentiment&source=states::states&pos=110&brush=28,31&queried=true&ex_cells= http://lstm.seas.harvard.edu/client/pattern_finder.html?data...

mrfusion 10y ago

why did they do this character by character? Would word by word make sense? Other than punctuation I'm not seeing why specific characters are meaningful units.

tshadley 10y ago
Word by word would require adding prior knowledge of words into the system, and they're trying to "start from scratch" as much as possible.

grandalf 10y ago

This has amazing potential for use in sock puppet accounts.

tshadley 10y ago

"The selected model reaches 1.12 bits per byte." (https://arxiv.org/pdf/1704.01444.pdf https://arxiv.org/pdf/1704.01444.pdf) For context, Claude Shannon found that humans could model English text with an entropy of 0.6 to 1.3 bits per character (http://languagelog.ldc.upenn.edu/myl/Shannon1950.pdf http://languagelog.ldc.upenn.edu/myl/Shannon1950.pdf)

djangowithme 10y ago

Why is the linear combination used to train the sentiment classifier? Why does its result get taken into account? Is this linear combination between 2 different strings?

du_bing 10y ago

Train on character-by-character basis, this is really incredible, quite opposite to human's intuition about language, but it seems a brilliant idea, and OpenAI tried it out, great!

andreyk 10y ago

I think it's fair to criticize this blog post for being unclear on what exactly is novel here; pre-training is a straighforward and old idea, but the blog post does not even mention this. Having accessible write ups for AI work is great, but surely it should not be confusing to domain experts or be written in such a way as to exacerbate the rampant oversimplification or misreporting in popular press about AI. Still, it is a cool mostly-experimental/empirical result, and it's good that these blog posts exist these days. For what it's worth, the paper predictably does a better job of covering the previous work and stating what their motivation was: "The experimental and evaluation protocols may be underestimating the quality of unsupervised representation learning for sentences and documents due to certain seemingly insignificant design decisions. Hill et al. (2016) also raises concern about current evaluation tasks in their recent work which provides a thorough survey of architectures and objectives for learning unsupervised sentence representations - including the above mentioned skip-thoughts. In this work, we test whether this is the case. We focus in on the task of sentiment analysis and attempt to learn an unsupervised representation that accurately contains this concept. Mikolov et al. (2013) showed that word-level recurrent language modelling supports the learning of useful word vectors and we are interested in pushing this line of work. As an approach, we consider the popular research benchmark of byte (character) level language modelling due to its further simplicity and generality. We are also interested in evaluating this approach as it is not immediately clear whether such a low-level training objective supports the learning of high-level representations." So, they question some built in assumptions from the past by training on lower-level data (characters), with a bigger dataset and more varied evaluation. The interesting result they highlight is that a single model unit is able to perform so well with their representation: "It is an open question why our model recovers the concept of sentiment in such a precise, disentangled, interpretable, and manipulable way. It is possible that sentiment as a conditioning feature has strong predictive capability for language modelling. This is likely since sentiment is such an important component of a review" , which I tend to agree with... train a on a whole lot of reviews, it's only natural to train a regressor for review sentiment.

kamalbanga 10y ago

What they have done is semi-supervised learning (Char-RNN) + supervised training of sentiment. Another way to do is semi-supervised learning (Word2Vec) + supervised training of sentiment. If first approach works better, does it imply that character level learning is more performant than word level learning?

eanzenberg 10y ago

I think one of the most amazing parts of this is how accessible the hardware is right now. You can get world-class AI results with the cost of less than most used cars. In addition, with so many resources freely available through open-source, the ability to get started is very accessible.

stillsut 10y ago

> The model struggles the more the input text diverges from review data This is where I fear the results will fail to scale. The ability to represent 'sentiment' as one neuron, and its ground truth as uni-dimensional seems most true to corpuses of online reviews where the entire point is to communicate whether you're happy with the thing that came out of the box. Most other forms of writing communicate sentiment in a more multi-dimensional way, and the subject of sentiment is more varied than a single item shipped in a box. In otherwords, the unreasonable simplicity of modelling a complex feature like sentiment with this method, is something of an artifact of this dataset.

mdibaiee 10y ago

As far as I understand, it means that there must be a relation between a character's sentiment and what the next character can (/should) be for neural network to use this as a feature, am I right? Does this mean we have unconsciously developed a language that exposes such relations?

deleted 10y ago
[deleted]
vhold 10y ago
They muse about the reason behind the sentiment neuron in the paper. "It is an open question why our model recovers the concept of sentiment in such a precise, disentangled, interpretable, and manipulable way. It is possible that sentiment as a conditioning feature has strong predictive capability for language modelling. This is likely since sentiment is such an important component of a review." They go on to frame that as an important consideration for further work like this: "Our work highlights the sensitivity of learned representations to the data distribution they are trained on. The results make clear that it is unrealistic to expect a model trained on a corpus of books, where the two most common genres are Romance and Fantasy, to learn an encoding which preserves the exact sentiment of a review." I'm wondering if a "funniness" neuron could be discovered in a model trained on millions of jokes of various funniness, or what sorts of undiscovered meaning there is in other neurons in this model.