3 ms·
Here's my first thought when I saw the visual input mode: Can I stream it screenshots, give it a goal, and ask it for click coordinates back? Can GPT-4 success
by sho_hn 4y ago
Here's my first thought when I saw the visual input mode:
Can I stream it screenshots, give it a goal, and ask it for click coordinates back? Can GPT-4 successfully use a PC desktop?
And my next thought:
Better try this with a VM without internet access ...
- polishdude20 4y agoRuneScape mining bot coming right up!
- galaxytachyon 4y agoThat is PaLM-E model. Or "embodied" LLM. They literally did exactly what you said. An LLM getting input from a camera and output command to a robotic hand. Then you can tell the bot to pick up a knife and "deposit" it somewhere else... Well, no theyjust made the bot picked up a pen and bring it to a person at a table 15ft away. It did that, despite "obstacles" by another human keep knocking the pen off its hand. But it shows that an image LLM can perform goals.