5 ms·

Some really interesting work lately on "contrastive" learning, where the accuracy is really getting on par with supervised learning, e.g. https://arxiv.org/abs/

Some really interesting work lately on "contrastive" learning, where the accuracy is really getting on par with supervised learning, e.g. https://arxiv.org/abs/2002.05709 https://arxiv.org/abs/2002.05709

jjoonathan 7y ago
For those of us out of the loop, could you summarize the idea of contrastive learning as a whole?
- yazr 7y ago
  A fully illustrated article [1] And Lilian Weng blog on self-supervision [2] .. CPC is .. translating a generative modeling problem to a classification problem... uses cross-entropy loss to measure how well the model can classify the “future” representation amongst a set of unrelated “negative” samples... [1] https://ankeshanand.com/blog/2020/01/26/contrative-self-supervised-learning.html https://ankeshanand.com/blog/2020/01/26/contrative-self-supe... [2] https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html#contrastive-predictive-coding https://lilianweng.github.io/lil-log/2019/11/10/self-supervi...
  - jjoonathan 7y ago
    Thanks!
- Voloskaya 7y ago
  (one variant of) The task is: Given a crop of an image (or a short audio snippet etc.), can you find the matching crop that also comes from the same image from a set containing a lot of negative samples (crops of others, unrelated images)? To succeed, the encoder needs to be able to extract the underlying, useful information (called slow features) contained in the patch and discard the noise as this will make the retrieval process much easier. This yields an encoder that gives pretty good representations of your inputs and you can then finetune some additional layers on top of it for your final task.
amitness 7y ago
Yup, this paper is really interesting. I'm reading it currently and will be posting an illustrated post soon on it.
- amitness 7y ago
  Here is the illustrated post on SimCLR: https://amitness.com/2020/03/illustrated-simclr/ https://amitness.com/2020/03/illustrated-simclr/

jjoonathan 7y ago

For those of us out of the loop, could you summarize the idea of contrastive learning as a whole?

yazr 7y ago
A fully illustrated article [1] And Lilian Weng blog on self-supervision [2] .. CPC is .. translating a generative modeling problem to a classification problem... uses cross-entropy loss to measure how well the model can classify the “future” representation amongst a set of unrelated “negative” samples... [1] https://ankeshanand.com/blog/2020/01/26/contrative-self-supervised-learning.html https://ankeshanand.com/blog/2020/01/26/contrative-self-supe... [2] https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html#contrastive-predictive-coding https://lilianweng.github.io/lil-log/2019/11/10/self-supervi...
- jjoonathan 7y ago
  Thanks!
Voloskaya 7y ago
(one variant of) The task is: Given a crop of an image (or a short audio snippet etc.), can you find the matching crop that also comes from the same image from a set containing a lot of negative samples (crops of others, unrelated images)? To succeed, the encoder needs to be able to extract the underlying, useful information (called slow features) contained in the patch and discard the noise as this will make the retrieval process much easier. This yields an encoder that gives pretty good representations of your inputs and you can then finetune some additional layers on top of it for your final task.

amitness 7y ago

Yup, this paper is really interesting. I'm reading it currently and will be posting an illustrated post soon on it.

amitness 7y ago
Here is the illustrated post on SimCLR: https://amitness.com/2020/03/illustrated-simclr/ https://amitness.com/2020/03/illustrated-simclr/