6 ms·

Well, by having a hand, and having a pen in it

by mjburgess 2y ago

idiotsecant 2y ago
I see. Your sense of sight is in some sense 'true' in a way that a webcam feed is not?
- mjburgess 2y ago
  Well if you can show me an LLM responding to it's having a pen in its hand via a robotic hand, webcam and the like -- then we are at the bare minimum for it possibly meaning, "i have a pen in my hand". No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) -- since, at least, no such corpus exists
  - magicalhippo 2y ago
    > No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) Seems we might not be that far away given the work on action tokens[1] and such. NVIDIA divoted a lot of the CES presentation to this kind of stuff[2]. [1]: https://arxiv.org/abs/2403.19578 https://arxiv.org/abs/2403.19578 (semi-random example) [2]: https://www.nvidia.com/en-us/ai/cosmos/ https://www.nvidia.com/en-us/ai/cosmos/
  - idiotsecant 2y ago
    Its pretty simple right now to present an llm with a text interface, a list of commands (turn head left turn head right, open hand, close hand, etc. And request they use those commands to achieve a goal. They can also almost all interpret images now. If I tell an llm that its objective is to look around until it finds its hand and tell me if its holding a pen or not, is that not exactly what you're talking about? Every single step there is well within the grasp of even the less advanced multimodal llms.
    - JohnKemeny 2y ago
      Until it sees the pen upside down and identifies it as a parachute. If there's a reasonable chance that this robot, when holding a pen, utters I have a parachute in my hand, does it then mean it? Does it know it?
  - nmstoker 2y ago
    How about Vision Language Models (VLMs)? They could easily take in a camera signal plus some basic hand sensor state and give structured output that would fit the bill here. https://huggingface.co/blog/vlms https://huggingface.co/blog/vlms

Well, by having a hand, and having a pen in it

idiotsecant 2y ago

I see. Your sense of sight is in some sense 'true' in a way that a webcam feed is not?

mjburgess 2y ago
Well if you can show me an LLM responding to it's having a pen in its hand via a robotic hand, webcam and the like -- then we are at the bare minimum for it possibly meaning, "i have a pen in my hand". No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) -- since, at least, no such corpus exists
- magicalhippo 2y ago
  > No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) Seems we might not be that far away given the work on action tokens[1] and such. NVIDIA divoted a lot of the CES presentation to this kind of stuff[2]. [1]: https://arxiv.org/abs/2403.19578 https://arxiv.org/abs/2403.19578 (semi-random example) [2]: https://www.nvidia.com/en-us/ai/cosmos/ https://www.nvidia.com/en-us/ai/cosmos/
- idiotsecant 2y ago
  Its pretty simple right now to present an llm with a text interface, a list of commands (turn head left turn head right, open hand, close hand, etc. And request they use those commands to achieve a goal. They can also almost all interpret images now. If I tell an llm that its objective is to look around until it finds its hand and tell me if its holding a pen or not, is that not exactly what you're talking about? Every single step there is well within the grasp of even the less advanced multimodal llms.
  - JohnKemeny 2y ago
    Until it sees the pen upside down and identifies it as a parachute. If there's a reasonable chance that this robot, when holding a pen, utters I have a parachute in my hand, does it then mean it? Does it know it?
- nmstoker 2y ago
  How about Vision Language Models (VLMs)? They could easily take in a camera signal plus some basic hand sensor state and give structured output that would fit the bill here. https://huggingface.co/blog/vlms https://huggingface.co/blog/vlms