3 ms·

Thanks for this writeup! Software engineers should definitely learn about Transformers. They are a significant step-change in the advancement of deep learning.

Thanks for this writeup! Software engineers should definitely learn about Transformers. They are a significant step-change in the advancement of deep learning. They've proven useful in multiple domains outside of natural language, including computer vision and audio. This opens up the world of multi-modal models which represent concepts by considering multiple types of input. The most popular one has been OpenAI's CLIP [0] If anyone's interested, I run a podcast and did an episode on Transformer models and their implications [1]. Check it out! [0] https://openai.com/blog/clip/ https://openai.com/blog/clip/ [1] https://www.youtube.com/watch?v=Kb0II5DuDE0 https://www.youtube.com/watch?v=Kb0II5DuDE0

dontreact 5y ago
Just a fun technical note: CLIP would be possible with any language and vision encoder. It turns out that transformers were the most efficient encoder choices available at the time but most likely the approach would have given interesting results with a resnet and some sort of convolutional language encoder. In fact the paper has a resnet as one model for the vision side.
- ShamelessC 5y ago
  Yeah they trained multiple ResNet models. The ViT's (vision transformers) tended to outperform the ResNet's although they discuss a wide range of datasets and I believe the ResNet ones are better in some cases. They have also been used for visual salience tasks as they may have a better representation of positional information in a scene. I think they do use a modified ResNet with some form of attention in it - but I'm not clear on the specifics and my understanding is that is somewhat common now.
amelius 5y ago
The Limitations section of [0] isn't very encouraging. > While CLIP usually performs well on recognizing common objects, it struggles on more abstract or systematic tasks such as counting the number of objects in an image and on more complex tasks such as predicting how close the nearest car is in a photo. On these two datasets, zero-shot CLIP is only slightly better than random guessing. Zero-shot CLIP also struggles compared to task specific models on very fine-grained classification, such as telling the difference between car models, variants of aircraft, or flower species. > CLIP also still has poor generalization to images not covered in its pre-training dataset. For instance, although CLIP learns a capable OCR system, when evaluated on handwritten digits from the MNIST dataset, zero-shot CLIP only achieves 88% accuracy, well below the 99.75% of humans on the dataset. Finally, we’ve observed that CLIP’s zero-shot classifiers can be sensitive to wording or phrasing and sometimes require trial and error “prompt engineering” to perform well.

dontreact 5y ago

Just a fun technical note: CLIP would be possible with any language and vision encoder. It turns out that transformers were the most efficient encoder choices available at the time but most likely the approach would have given interesting results with a resnet and some sort of convolutional language encoder. In fact the paper has a resnet as one model for the vision side.

ShamelessC 5y ago
Yeah they trained multiple ResNet models. The ViT's (vision transformers) tended to outperform the ResNet's although they discuss a wide range of datasets and I believe the ResNet ones are better in some cases. They have also been used for visual salience tasks as they may have a better representation of positional information in a scene. I think they do use a modified ResNet with some form of attention in it - but I'm not clear on the specifics and my understanding is that is somewhat common now.

amelius 5y ago

The Limitations section of [0] isn't very encouraging. > While CLIP usually performs well on recognizing common objects, it struggles on more abstract or systematic tasks such as counting the number of objects in an image and on more complex tasks such as predicting how close the nearest car is in a photo. On these two datasets, zero-shot CLIP is only slightly better than random guessing. Zero-shot CLIP also struggles compared to task specific models on very fine-grained classification, such as telling the difference between car models, variants of aircraft, or flower species. > CLIP also still has poor generalization to images not covered in its pre-training dataset. For instance, although CLIP learns a capable OCR system, when evaluated on handwritten digits from the MNIST dataset, zero-shot CLIP only achieves 88% accuracy, well below the 99.75% of humans on the dataset. Finally, we’ve observed that CLIP’s zero-shot classifiers can be sensitive to wording or phrasing and sometimes require trial and error “prompt engineering” to perform well.