4 ms·
It takes more than that. Put a LLM in a robot and it’ll still lie to you because it has no idea what anything is or means.
by more_corn 4y ago
It takes more than that. Put a LLM in a robot and it’ll still lie to you because it has no idea what anything is or means.
- heyitsguay 4y agoYeah, but build a new model that injects camera and microphone observations as well as robotic hardware internal state estimates into the same embedding space as your favorite LLM, fine tune the whole thing, and suddenly your text generator is referencing a semantic space that coreferences observed physical relationships. You break out of the circular reference loop pure LLMs suffer from. Embodiment and grounding all at once. Google's recent PaLM-E paper shows how much progress is already being made towards exactly this sort of multimodal embedding soup. Things are going to keep getting wilder!