Yeah, but build a new model that injects camera and microphone observations as well as robotic hardware internal state estimates into the same embedding space a

Yeah, but build a new model that injects camera and microphone observations as well as robotic hardware internal state estimates into the same embedding space as your favorite LLM, fine tune the whole thing, and suddenly your text generator is referencing a semantic space that coreferences observed physical relationships. You break out of the circular reference loop pure LLMs suffer from. Embodiment and grounding all at once. Google's recent PaLM-E paper shows how much progress is already being made towards exactly this sort of multimodal embedding soup. Things are going to keep getting wilder!