Google actually recently went some steps further and combined the PaLM LLM (bigger than GPT-3.5) with a 22 billion parameter Vision Transformer to do this - ht

Google actually recently went some steps further and combined the PaLM LLM (bigger than GPT-3.5) with a 22 billion parameter Vision Transformer to do this - https://palm-e.github.io https://palm-e.github.io