That is PaLM-E model. Or "embodied" LLM. They literally did exactly what you said. An LLM getting input from a camera and output command to a robotic hand. Then

That is PaLM-E model. Or "embodied" LLM. They literally did exactly what you said. An LLM getting input from a camera and output command to a robotic hand. Then you can tell the bot to pick up a knife and "deposit" it somewhere else... Well, no theyjust made the bot picked up a pen and bring it to a person at a table 15ft away. It did that, despite "obstacles" by another human keep knocking the pen off its hand. But it shows that an image LLM can perform goals.