Something I've noticed is that there's a ton of work going into better vision transformers, but whenever there's a big new multimodal result or something, every

Something I've noticed is that there's a ton of work going into better vision transformers, but whenever there's a big new multimodal result or something, everyone's just using ViT. What gives? Like in CLIP or PaLM-E, they very easily could have used another vision architecture, it's not like they were running all modalities through the same encoder.