6 ms·

I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled exam

I don't know, but this seems a bit hyped in places. They start with: > Our L1-regularized model matches multichannel CNN performance with only 11 labeled examples, and state-of-the-art CT-LSTM Ensembles with 232 examples. Hmm, that sounds pretty impressive. But then later you read: > We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chunk of text. Training took one month across four NVIDIA Pascal GPUs Wait, what? How did "232 examples" transform into "82 million"?? OK, I get it: they pretrained the network on the 82M reviews, and then trained the last layer to do the sentiment analysis. But you can't honestly claim that you did great with just 232 examples!

gdb 10y ago
Thanks for the feedback — added context to that sentence to make it more clear! The main interesting thing is that none of the Amazon data was labeled, while the 232 labeled examples were.
- p1esk 10y ago
  Very interesting, this reminds me of the 2012 paper by Andrew Ng: Building High-level Features Using Large Scale Unsupervised Learning
- joshuamorton 10y ago
  To further clarify: does unlabeled mean "we didn't use sentiment data" or "we were only trying to predict the next character given the prior characters", since the amazon data does come with associated 1-5 star ratings, were those used or not?
  - gdb 10y ago
    We did not use the star ratings.
    - joshuamorton 10y ago
      That's what I thought, and that makes this all the more interesting!
    - technotony 10y ago
      Have you looked at cross referencing against the star ratings? Would be interesting to see how predictive this was of those ratings... or even if another variable in the system could predict that?
tensor 10y ago
I think it's a fair claim. Labelled data is very hard to come by compared to unlabelled data. Being able to get a highly accurate model with only a small amount of labelled data is a very sought after and practical property.
- emcq 10y ago
  The technique of training a model on a lot of data for a long time and then leveraging its sophisticated representation only learning the last layer(s) on small datasets to create accurate models is common practice.
  - rspeer 10y ago
    It's a good idea, but it doesn't seem very common so far. This is what my NLP company (Luminoso) does -- we train a domain-general model of word meanings on a lot of data, then do the last step on the probably-small amount of specific data you actually have. Even customers who are knowledgeable about machine learning usually haven't heard of the idea before. They've been assuming that the only way to do NLP is to get millions of labeled examples. Or to get a thousand labeled examples and put them into the kind of off-the-shelf algorithm that needs millions of labeled examples, which of course goes poorly.
    - argonaut 10y ago
      My ML background is primarily undergraduate work, and even then it was extremely common.
  - halflings 10y ago
    It's not as easy as what you just described, especially on sequential data. Sure, people already use embeddings built by different models for different tasks (think word2vec, last layer from inception, etc.) but this rarely performs as well as what this article shows.
  - Danylon 10y ago
    Beating the state-of-the-art with one-shot learning is not common. Transfer learning for NLP is also quite unchartered. Also the technique is quite novel: This is not pre-trained nets on labeled data, it is an unsupervised generative model. Future research directions are exciting: Unsupervised prediction of the next frame in a video, and then being able to one-shot learn a wide range of visual tasks.
dnautics 10y ago
Is this correct? My sense from the article is that they did all the training on unsupervised, and then checked one of the recurrent lines for a correlation to sentiment, not adding a layer and doing more training.
derefr 10y ago
This actually demonstrates something very interesting, I think: you can take an ML model trained with the "low-level prerequisite knowledge" of a subject, and then very quickly and easily teach it a high-level concept that relies on that knowledge. Which, now that I think about it, makes the human brain and its amazing adaptive general-game-playing abilities a bit less mysterious. Since we humans all have these huge corpuses of sense-data we've been receiving reinforcement signals about since birth, we've likely built up all sorts of low-level models which we just use to predict the world for reflex responses a little bit better and faster (speech models so we can respond to what people are saying even as they're still saying it, visual models so we can throw spears where lions are going to be instead of where they are, etc.) But those low-level predictive models make it nearly effortless to build higher-level models. I wonder if we'd take a giant leap forward in AI if we just managed to scan+emulate a regular animal brain (say, of a rat), and then built the AI as a neocortex-equivalent for that brain. It would have instant access to thousands or millions of pre-trained low-level predictive models, which it could easily discover as having outputs correlated to success and thus "attach to" during its own training.
- emcq 10y ago
  What you describe is exactly what practitioners in the field have been doing for years. I think that's why the parent is a bit puzzled at the publication, as it's difficult to understand what's novel.
  - joshuamorton 10y ago
    Indeed, although I'm not as familiar with transfers this far (that is, fine-tuning models is really common with image recognition tasks, but often you take a model trained to recognize objects in images, and train it to recognize different objects in images, but you're still recognizing objects in images). This feels a bit different, in that yesterday I would have had no strong intuition that "a char-rnn can detect text sentiment better than sota". Looking now, I can rationalize that idea. I get why it might make sense, but it was non-obvious. Do you disagree with any of that? (and if you do, I'd love to see this distance of transfer in literature, its always cool to read up on these things)
sunrisetofu 10y ago
It's essentially the nlp equivalent transfer learning from the computer vision side.
- akyu 10y ago
  No this is quite a bit different. In transfer learning in computer vision, people are using a network trained on another, typically large dataset, and using the learned features on a different data set. This LSTM simply learned to predict the data. It didn't learn some other supervised task. This would be more similar to autoencoder pretraining, but even that is not quite the same.
gabrielgoh 10y ago
They can - because the examples weren't labelled. In principle, you could use the model to train a whole lot of other things (what age group does a product appeal to? Is this product targeted at men/women?) with just a few hundred examples.
jorgemf 10y ago
isn't it impressive? They trained a network to predict the next character and when the finished they trained the last layer to predict sentiment with only few examples. The network was able to learn sentiment somehow in the first training phase without telling it! For me this open my mind to new opportunities when training deep learning. For example I can do the same for images: train a network to recognize objects and later use the same network to predict sentiment or prettiness for example. And the best thing is that I don't need a lot of labeled examples for the second phase of training!
- taneq 10y ago
  Immediate thought - the network didn't learn sentiment in the training phase, it just clustered the data. The last phase was so quick to train because it was labeling already-clustered data, so a few data points in each cluster was enough to make it 'obvious'.
  - jorgemf 10y ago
    Actually it was only an output neuron, and for me it is the same if you call it cluster or not. It just learnt sentiment without any hint. Now I wonder if they can train it again and get the same sentiment neuron, that would mean something. Like sentiment is necessary to predict the next character? (word?)
mkagenius 10y ago
With 82 million examples it learned how to spell each word and which word follows which in a sentence. With the 232 examples it learned What bad sentiment sentences look like and which words occur in them.

gdb 10y ago

Thanks for the feedback — added context to that sentence to make it more clear! The main interesting thing is that none of the Amazon data was labeled, while the 232 labeled examples were.

p1esk 10y ago
Very interesting, this reminds me of the 2012 paper by Andrew Ng: Building High-level Features Using Large Scale Unsupervised Learning
joshuamorton 10y ago
To further clarify: does unlabeled mean "we didn't use sentiment data" or "we were only trying to predict the next character given the prior characters", since the amazon data does come with associated 1-5 star ratings, were those used or not?
- gdb 10y ago
  We did not use the star ratings.
  - joshuamorton 10y ago
    That's what I thought, and that makes this all the more interesting!
  - technotony 10y ago
    Have you looked at cross referencing against the star ratings? Would be interesting to see how predictive this was of those ratings... or even if another variable in the system could predict that?

tensor 10y ago

I think it's a fair claim. Labelled data is very hard to come by compared to unlabelled data. Being able to get a highly accurate model with only a small amount of labelled data is a very sought after and practical property.

emcq 10y ago
The technique of training a model on a lot of data for a long time and then leveraging its sophisticated representation only learning the last layer(s) on small datasets to create accurate models is common practice.
- rspeer 10y ago
  It's a good idea, but it doesn't seem very common so far. This is what my NLP company (Luminoso) does -- we train a domain-general model of word meanings on a lot of data, then do the last step on the probably-small amount of specific data you actually have. Even customers who are knowledgeable about machine learning usually haven't heard of the idea before. They've been assuming that the only way to do NLP is to get millions of labeled examples. Or to get a thousand labeled examples and put them into the kind of off-the-shelf algorithm that needs millions of labeled examples, which of course goes poorly.
  - argonaut 10y ago
    My ML background is primarily undergraduate work, and even then it was extremely common.
- halflings 10y ago
  It's not as easy as what you just described, especially on sequential data. Sure, people already use embeddings built by different models for different tasks (think word2vec, last layer from inception, etc.) but this rarely performs as well as what this article shows.
- Danylon 10y ago
  Beating the state-of-the-art with one-shot learning is not common. Transfer learning for NLP is also quite unchartered. Also the technique is quite novel: This is not pre-trained nets on labeled data, it is an unsupervised generative model. Future research directions are exciting: Unsupervised prediction of the next frame in a video, and then being able to one-shot learn a wide range of visual tasks.

dnautics 10y ago

Is this correct? My sense from the article is that they did all the training on unsupervised, and then checked one of the recurrent lines for a correlation to sentiment, not adding a layer and doing more training.

derefr 10y ago

This actually demonstrates something very interesting, I think: you can take an ML model trained with the "low-level prerequisite knowledge" of a subject, and then very quickly and easily teach it a high-level concept that relies on that knowledge. Which, now that I think about it, makes the human brain and its amazing adaptive general-game-playing abilities a bit less mysterious. Since we humans all have these huge corpuses of sense-data we've been receiving reinforcement signals about since birth, we've likely built up all sorts of low-level models which we just use to predict the world for reflex responses a little bit better and faster (speech models so we can respond to what people are saying even as they're still saying it, visual models so we can throw spears where lions are going to be instead of where they are, etc.) But those low-level predictive models make it nearly effortless to build higher-level models. I wonder if we'd take a giant leap forward in AI if we just managed to scan+emulate a regular animal brain (say, of a rat), and then built the AI as a neocortex-equivalent for that brain. It would have instant access to thousands or millions of pre-trained low-level predictive models, which it could easily discover as having outputs correlated to success and thus "attach to" during its own training.

emcq 10y ago
What you describe is exactly what practitioners in the field have been doing for years. I think that's why the parent is a bit puzzled at the publication, as it's difficult to understand what's novel.
- joshuamorton 10y ago
  Indeed, although I'm not as familiar with transfers this far (that is, fine-tuning models is really common with image recognition tasks, but often you take a model trained to recognize objects in images, and train it to recognize different objects in images, but you're still recognizing objects in images). This feels a bit different, in that yesterday I would have had no strong intuition that "a char-rnn can detect text sentiment better than sota". Looking now, I can rationalize that idea. I get why it might make sense, but it was non-obvious. Do you disagree with any of that? (and if you do, I'd love to see this distance of transfer in literature, its always cool to read up on these things)

sunrisetofu 10y ago

It's essentially the nlp equivalent transfer learning from the computer vision side.

akyu 10y ago
No this is quite a bit different. In transfer learning in computer vision, people are using a network trained on another, typically large dataset, and using the learned features on a different data set. This LSTM simply learned to predict the data. It didn't learn some other supervised task. This would be more similar to autoencoder pretraining, but even that is not quite the same.

gabrielgoh 10y ago

They can - because the examples weren't labelled. In principle, you could use the model to train a whole lot of other things (what age group does a product appeal to? Is this product targeted at men/women?) with just a few hundred examples.

jorgemf 10y ago

isn't it impressive? They trained a network to predict the next character and when the finished they trained the last layer to predict sentiment with only few examples. The network was able to learn sentiment somehow in the first training phase without telling it! For me this open my mind to new opportunities when training deep learning. For example I can do the same for images: train a network to recognize objects and later use the same network to predict sentiment or prettiness for example. And the best thing is that I don't need a lot of labeled examples for the second phase of training!

taneq 10y ago
Immediate thought - the network didn't learn sentiment in the training phase, it just clustered the data. The last phase was so quick to train because it was labeling already-clustered data, so a few data points in each cluster was enough to make it 'obvious'.
- jorgemf 10y ago
  Actually it was only an output neuron, and for me it is the same if you call it cluster or not. It just learnt sentiment without any hint. Now I wonder if they can train it again and get the same sentiment neuron, that would mean something. Like sentiment is necessary to predict the next character? (word?)

mkagenius 10y ago

With 82 million examples it learned how to spell each word and which word follows which in a sentence. With the 232 examples it learned What bad sentiment sentences look like and which words occur in them.