2 ms·
At a minimum he glossed over the multimodal capabilities of GPT-4. If they use the same set of tokens, it’s unclear how this doesn’t pollute text training data.
by janalsncm 3y ago
At a minimum he glossed over the multimodal capabilities of GPT-4. If they use the same set of tokens, it’s unclear how this doesn’t pollute text training data. If they use separate tokens, the model size should be bigger.
- ShamelessC 3y agoI assume they just intersperse and/or use a special <IMAGE> token to prime the models appropriately. This has been done before by others training on e.g. HTML with <img> tags replaced with image tokens from a vision model.
- chriskanan 3y agoThis is the same strategy used by Palm-E and Mini-GPT4. They have a projection layer that transforms the vision embeddings into the token embedding space.