This model is vision-only so it can't be SOTA even if it's #1 performing in many of the original categories of benchmarks, which it is (it's a very very good mo

This model is vision-only so it can't be SOTA even if it's #1 performing in many of the original categories of benchmarks, which it is (it's a very very good model). We've moved on from ImageNet-style tests "Choose the most appropriate label for this image from 200 possible labels" to much more advanced "Reasoning" tests[0]. PaLI[1] is potentially the SoTA here but BeIT-3[2] may be better example for my thesis. Notice that BeIT-3 is trained on not just images, but also trained in natural language. It outperforms purely image-trained models on even pure-image tasks like Object Detection and Semantic Segmentation. Take a look at the major benchmarks for Segmentation (ade20k) [4]: DINOv2, 11th place. BEiT-3, 4th place. Yes, BEiT-3 has 72% more parameters but it's also basically an entire LLM. Even GPT-4 is a multi-modal model, and actually accepts images as prompt inputs, OpenAI just doesn't expose that ability. More importantly, the new multi-modal models can understand human questioning like "What type of flowers are in the blue buckets of this image?" and respond intelligently, in English/whatever. DINOv2 was trained with techniques borrowed from LLM training methods, but is not trained for natural language. 0: https://paperswithcode.com/area/reasoning https://paperswithcode.com/area/reasoning 1: https://arxiv.org/pdf/2209.06794v2.pdf https://arxiv.org/pdf/2209.06794v2.pdf 2: https://paperswithcode.com/paper/image-as-a-foreign-language-beit-pretraining https://paperswithcode.com/paper/image-as-a-foreign-language... 3: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html 4: https://paperswithcode.com/sota/semantic-segmentation-on-ade20k https://paperswithcode.com/sota/semantic-segmentation-on-ade...