3 ms·
Wow. So far this seems better than mini-GPT4. The most impressive thing is how little training data this took. and the fact that you only need CLIP and a linear
by famouswaffles 4y ago
Wow. So far this seems better than mini-GPT4. The most impressive thing is how little training data this took. and the fact that you only need CLIP and a linear projection is wild. at least with mini-GPT-4, you could say that BLIP-2 already translates RGB into text token space and maybe a simple linear projection should work. But that you can project the image encoding directly...wild.
EDIT - Just found this paper that demonstrated this a few months ago (that somehow language and vision models learn representations similar enough that linear projection is enough) https://arxiv.org/abs/2209.15162 https://arxiv.org/abs/2209.15162
- MacsHeadroom 4y ago> The most impressive thing is how little training data this took. Yes, this is just the beginning of multi-modality. Like how Alpaca was quickly overshadowed with much better finetunes with larger and improved datasets, I expect we'll see more capable (and truly open source) multi-modal models in the near future. Interestingly, this "billion-scale corpus of images interleaved with text" was just open sourced a few days ago: https://github.com/allenai/mmc4 https://github.com/allenai/mmc4 Looks like low hanging fruit for any team who wants to attempt to become the "Vicuna" of multi-modal transformers for a while!
- famouswaffles 4y agoI feel like i discovered a deep truth today. People say we need to ground language models but it turns that they're already being grounded. Text-only (before the multimodality) GPT-4's visual and spatial reasoning make so much more sense now.