2 ms·

In what way do you have access to the truth conditions for 'I have a pen in my hand' that an LLM can not? This smells circular to me.

deleted 2y ago
[deleted]
mjburgess 2y ago
Well, by having a hand, and having a pen in it
- idiotsecant 2y ago
  I see. Your sense of sight is in some sense 'true' in a way that a webcam feed is not?
  - mjburgess 2y ago
    Well if you can show me an LLM responding to it's having a pen in its hand via a robotic hand, webcam and the like -- then we are at the bare minimum for it possibly meaning, "i have a pen in my hand". No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) -- since, at least, no such corpus exists
    - magicalhippo 2y ago
      > No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) Seems we might not be that far away given the work on action tokens[1] and such. NVIDIA divoted a lot of the CES presentation to this kind of stuff[2]. [1]: https://arxiv.org/abs/2403.19578 https://arxiv.org/abs/2403.19578 (semi-random example) [2]: https://www.nvidia.com/en-us/ai/cosmos/ https://www.nvidia.com/en-us/ai/cosmos/
    - idiotsecant 2y ago
      Its pretty simple right now to present an llm with a text interface, a list of commands (turn head left turn head right, open hand, close hand, etc. And request they use those commands to achieve a goal. They can also almost all interpret images now. If I tell an llm that its objective is to look around until it finds its hand and tell me if its holding a pen or not, is that not exactly what you're talking about? Every single step there is well within the grasp of even the less advanced multimodal llms.
      - JohnKemeny 2y ago
        Until it sees the pen upside down and identifies it as a parachute. If there's a reasonable chance that this robot, when holding a pen, utters I have a parachute in my hand, does it then mean it? Does it know it?
    - nmstoker 2y ago
      How about Vision Language Models (VLMs)? They could easily take in a camera signal plus some basic hand sensor state and give structured output that would fit the bill here. https://huggingface.co/blog/vlms https://huggingface.co/blog/vlms

deleted 2y ago

[deleted]

mjburgess 2y ago

Well, by having a hand, and having a pen in it

idiotsecant 2y ago
I see. Your sense of sight is in some sense 'true' in a way that a webcam feed is not?
- mjburgess 2y ago
  Well if you can show me an LLM responding to it's having a pen in its hand via a robotic hand, webcam and the like -- then we are at the bare minimum for it possibly meaning, "i have a pen in my hand". No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) -- since, at least, no such corpus exists
  - magicalhippo 2y ago
    > No such LLMs exist, because they are trained to predict the next word not (WebCamState, RobotArmState, NextWord) Seems we might not be that far away given the work on action tokens[1] and such. NVIDIA divoted a lot of the CES presentation to this kind of stuff[2]. [1]: https://arxiv.org/abs/2403.19578 https://arxiv.org/abs/2403.19578 (semi-random example) [2]: https://www.nvidia.com/en-us/ai/cosmos/ https://www.nvidia.com/en-us/ai/cosmos/
  - idiotsecant 2y ago
    Its pretty simple right now to present an llm with a text interface, a list of commands (turn head left turn head right, open hand, close hand, etc. And request they use those commands to achieve a goal. They can also almost all interpret images now. If I tell an llm that its objective is to look around until it finds its hand and tell me if its holding a pen or not, is that not exactly what you're talking about? Every single step there is well within the grasp of even the less advanced multimodal llms.
    - JohnKemeny 2y ago
      Until it sees the pen upside down and identifies it as a parachute. If there's a reasonable chance that this robot, when holding a pen, utters I have a parachute in my hand, does it then mean it? Does it know it?
  - nmstoker 2y ago
    How about Vision Language Models (VLMs)? They could easily take in a camera signal plus some basic hand sensor state and give structured output that would fit the bill here. https://huggingface.co/blog/vlms https://huggingface.co/blog/vlms