Since they're using LLama with Llava, they're getting lots of Llama's baked-in understanding of the world. And I believe that Llama had multimodal training data

Since they're using LLama with Llava, they're getting lots of Llama's baked-in understanding of the world. And I believe that Llama had multimodal training data as part of its pipeline, whether or not we get access to the heads that parse that data in the open weights. So, I'm saying that Llama has some ability to describe / complete captions of photos, and very probably that ability comes from some multimodal training done at FB, and that using it in this way to train something new that beats benchmarks is a form of pulling out the value of that training data. p.s. In the original comment, I meant to type 'multimodal' and didn't, or got autocorrected and didn't notice - apologies