Some good some bad here; it looks like you can in general get better descriptions out of LLama-3+Llava-1.5 than random crappy / short / possibly wrong human cap

Some good some bad here; it looks like you can in general get better descriptions out of LLama-3+Llava-1.5 than random crappy / short / possibly wrong human captions. The image generation models they train using the new caption system have better coherence and look a lot better / less weird. (I don’t know if they showed training detail differences between their comparisons, but let’s assume they’re not hiding the ball here). The bad: Those captions don’t always perform better on all tests using existing infra - some of their tests perform worse against existing benchmarks. Why they perform worse is an open question; the answer might be that the existing test data expects short / human answers, or it might be that the models are more verbose but less accurate (although that seems unlikely looking at the examples they show). Also troubling to me is a toss-off note in Table 3 showing that the “concat” data, one in which they use original caption concatenated to the generated caption, has really bad CLIP outcomes. That … surprises me based on the rest of the results. That said, their baseline comparison numbers in Table 1 are pretty compelling — this would definitely go into the ‘try it’ category based on the paper. But, it also goes into the ‘might fail in really surprising ways, so proceed carefully’ category for me. EDIT: Another way to think of this model is to say “can we exfiltrate multi-model training data from open weights / closed training models?” And I think there the answer is yes.