3 ms·

The technique of training a model on a lot of data for a long time and then leveraging its sophisticated representation only learning the last layer(s) on small

by emcq 10y ago

The technique of training a model on a lot of data for a long time and then leveraging its sophisticated representation only learning the last layer(s) on small datasets to create accurate models is common practice.

rspeer 10y ago
It's a good idea, but it doesn't seem very common so far. This is what my NLP company (Luminoso) does -- we train a domain-general model of word meanings on a lot of data, then do the last step on the probably-small amount of specific data you actually have. Even customers who are knowledgeable about machine learning usually haven't heard of the idea before. They've been assuming that the only way to do NLP is to get millions of labeled examples. Or to get a thousand labeled examples and put them into the kind of off-the-shelf algorithm that needs millions of labeled examples, which of course goes poorly.
- argonaut 10y ago
  My ML background is primarily undergraduate work, and even then it was extremely common.
halflings 10y ago
It's not as easy as what you just described, especially on sequential data. Sure, people already use embeddings built by different models for different tasks (think word2vec, last layer from inception, etc.) but this rarely performs as well as what this article shows.
Danylon 10y ago
Beating the state-of-the-art with one-shot learning is not common. Transfer learning for NLP is also quite unchartered. Also the technique is quite novel: This is not pre-trained nets on labeled data, it is an unsupervised generative model. Future research directions are exciting: Unsupervised prediction of the next frame in a video, and then being able to one-shot learn a wide range of visual tasks.
- kastnerkyle 10y ago
  You might be interested in minute 50 onward [0], or this recent paper from Facebook [1]. [0] https://www.youtube.com/watch?v=-yX1SYeDHbg&list=PLE6Wd9FR--EfW8dtjAuPoTuPcqmOV53Fu&index=4 https://www.youtube.com/watch?v=-yX1SYeDHbg&list=PLE6Wd9FR--... [1] https://arxiv.org/abs/1703.07684 https://arxiv.org/abs/1703.07684
  - Danylon 10y ago
    Cool. I knew of previous work [1], but not the recent paper you posted. Thanks. [1] https://arxiv.org/abs/1412.6056 https://arxiv.org/abs/1412.6056 "Predicting Deeper into the Future of Semantic Segmentation"

by emcq 10y ago

rspeer 10y ago

It's a good idea, but it doesn't seem very common so far. This is what my NLP company (Luminoso) does -- we train a domain-general model of word meanings on a lot of data, then do the last step on the probably-small amount of specific data you actually have. Even customers who are knowledgeable about machine learning usually haven't heard of the idea before. They've been assuming that the only way to do NLP is to get millions of labeled examples. Or to get a thousand labeled examples and put them into the kind of off-the-shelf algorithm that needs millions of labeled examples, which of course goes poorly.

argonaut 10y ago
My ML background is primarily undergraduate work, and even then it was extremely common.

halflings 10y ago

It's not as easy as what you just described, especially on sequential data. Sure, people already use embeddings built by different models for different tasks (think word2vec, last layer from inception, etc.) but this rarely performs as well as what this article shows.

Danylon 10y ago

Beating the state-of-the-art with one-shot learning is not common. Transfer learning for NLP is also quite unchartered. Also the technique is quite novel: This is not pre-trained nets on labeled data, it is an unsupervised generative model. Future research directions are exciting: Unsupervised prediction of the next frame in a video, and then being able to one-shot learn a wide range of visual tasks.

kastnerkyle 10y ago
You might be interested in minute 50 onward [0], or this recent paper from Facebook [1]. [0] https://www.youtube.com/watch?v=-yX1SYeDHbg&list=PLE6Wd9FR--EfW8dtjAuPoTuPcqmOV53Fu&index=4 https://www.youtube.com/watch?v=-yX1SYeDHbg&list=PLE6Wd9FR--... [1] https://arxiv.org/abs/1703.07684 https://arxiv.org/abs/1703.07684
- Danylon 10y ago
  Cool. I knew of previous work [1], but not the recent paper you posted. Thanks. [1] https://arxiv.org/abs/1412.6056 https://arxiv.org/abs/1412.6056 "Predicting Deeper into the Future of Semantic Segmentation"