4 ms·

If you upload pictures of every room in your house to an LLM it can definitely tell you where the dishwasher is. If your argument is just that they cant walk ar

If you upload pictures of every room in your house to an LLM it can definitely tell you where the dishwasher is. If your argument is just that they cant walk around your house so they cant be intelligent I think thats pretty clearly wrong.

kimixa 2y ago
A trained image recognition model could probably recognize a dishwasher from an image. But that won't be the same model that writes bad poetry or tries to autocomplete your next line of code. Or control the legs of a robot to move towards the dishwasher while holding a dirty plate. And each has a fair bit of manual tuning and preprocessing based on its function which may simply not be applicable to other areas even with scale. The best performing models aren't just taking in unstructured untyped data. Even the most flexible models are only tackling a small slice of what "intelligence" is.
- jdietrich 2y ago
  ChatGPT, Gemini and Claude are all natively multimodal. They can recognise a dishwasher from an image, among many other things. https://www.youtube.com/watch?v=KwNUJ69RbwY https://www.youtube.com/watch?v=KwNUJ69RbwY
  - cynicalpeace 2y ago
    Can they take the pictures?
    - ta988 2y ago
      Technically yes they can run functions. There were experiments of Claude used to run a robot around a house. So technically, we are not far at all and current models may even be able to do it.
      - cynicalpeace 2y ago
        Please re-read my original comment. "The AI systems need bodies to actually learn these things." I never said this was impossible to achieve.
        sippeangelo 2y ago
        Can your brain see the dishwasher without your eyes?
    - sdenton4 2y ago
      But do they have strong beaks? https://sktchd.com/column/comics-disassembled-ten-things-of-note-from-the-past-week-in-comics-led-by-boy-wonders/false-knees/ https://sktchd.com/column/comics-disassembled-ten-things-of-...
cynicalpeace 2y ago
Do they know what a hot shower feels like? They can describe it. But do they actually know? Have they experienced it? This is my point. Nerds keep dismissing physicality and experience. If your argument is a brain in a jar will be generally intelligent, I think that's pretty clearly wrong.
- HDThoreaun 2y ago
  See the responses section https://en.wikipedia.org/wiki/Knowledge_argument https://en.wikipedia.org/wiki/Knowledge_argument This idea certainly has been long considered but I personally reject it.
  - cynicalpeace 2y ago
    While interesting, this is a separate thought experiment with its own quirks. Sort of a strawman, since my argument is formulated differently and simply argues that AIs need to be more than brains in jars for them to be considered generally intelligent. And that the only reason we think AIs can just be brains in jars is because many of the people developing them consider themselves as simply brains in jars.
    - HDThoreaun 2y ago
      Not really. The point of it is considering whether physical experience creates knowledge that is impossible to get otherwise. Thats the argument you are making no? If Mary learns nothing new when seeing red for the first time an AI would also learn nothing new when seeing red for the first time. > Do they know what a hot shower feels like? They can describe it. But do they actually know? Have they experienced it Is directly a knowledge argument
      - cynicalpeace 2y ago
        Mary in that thought experiment is not an LLM that has learned via text. She's acquired "all the physical information there is to obtain about what goes on when we see ripe tomatoes". This does not actually describe modern LLMs. It actually better describes a robot that has transcribed the location, temperature, and velocity of water drops from a hot shower to its memory. Again, this thought experiment has its own quirks. Also, it is an argument against physicalism, which I have no interest in debating. While it's tangentially related, my point is not for/against physicalism. My argument is about modern AI and it's ability to learn. If we put touch sensors, eyes, nose, a mechanism to collect physical data (legs) and even sex organs on an AI system, then it is more generally intelligent than before. It will have learned in a better fashion what a hot shower feels like and will be smarter for it.
magpi3 2y ago
Could it tell the difference between a dishwasher and a picture of a dishwasher on a wall? Or one painted onto a wall? Or a toy dishwasher? There is an essential idea of what makes something a dishwasher that LLM's will never be able to grasp no matter how many models you throw at them. They would have to fundamentally understand that what they are "seeing" is an electronic appliance connected to the plumbing that washes dishes. The sound of a running dishwasher, the heat you feel when you open one, and the wet, clean dishes is also part of that understanding.
- viraptor 2y ago
  Yes, it can tell a difference, up to the point where the boundaries are getting fuzzy. But the same thing applies to us all. Can you tell this is a dishwasher? https://www.amazon.com.au/Countertop-Dishwasher-Automatic-Vegetable-Apartments/dp/B0CLKTCVFC https://www.amazon.com.au/Countertop-Dishwasher-Automatic-Ve... Can you tell this is a drawing of a glass? https://www.deviantart.com/januarysnow13/art/Wine-Glass-Hyperrealistic-Drawing-506809946 https://www.deviantart.com/januarysnow13/art/Wine-Glass-Hype... Can you tell this is a toy? https://www.amazon.com.au/Theo-Klein-Miele-Washing-Machine/dp/B00UCK33HU https://www.amazon.com.au/Theo-Klein-Miele-Washing-Machine/d...
  - magpi3 2y ago
    If I am limited to looking at pictures, then I am at the same disadvantage as the LLM, sure. The point is that people can experience and understand objects from a multitude of perspectives, both with our senses and the mental models we utilize to understand the object. Can LLMs do the same?
    - viraptor 2y ago
      That's not a disadvantage of LLM. You can start sending images from a camera moving around and you'll get many views as well. The capabilities here are the same as the eye-brain system - it can't move independently either.
      - 2y ago

kimixa 2y ago

A trained image recognition model could probably recognize a dishwasher from an image. But that won't be the same model that writes bad poetry or tries to autocomplete your next line of code. Or control the legs of a robot to move towards the dishwasher while holding a dirty plate. And each has a fair bit of manual tuning and preprocessing based on its function which may simply not be applicable to other areas even with scale. The best performing models aren't just taking in unstructured untyped data. Even the most flexible models are only tackling a small slice of what "intelligence" is.

jdietrich 2y ago
ChatGPT, Gemini and Claude are all natively multimodal. They can recognise a dishwasher from an image, among many other things. https://www.youtube.com/watch?v=KwNUJ69RbwY https://www.youtube.com/watch?v=KwNUJ69RbwY
- cynicalpeace 2y ago
  Can they take the pictures?
  - ta988 2y ago
    Technically yes they can run functions. There were experiments of Claude used to run a robot around a house. So technically, we are not far at all and current models may even be able to do it.
    - cynicalpeace 2y ago
      Please re-read my original comment. "The AI systems need bodies to actually learn these things." I never said this was impossible to achieve.
      - sippeangelo 2y ago
        Can your brain see the dishwasher without your eyes?
  - sdenton4 2y ago
    But do they have strong beaks? https://sktchd.com/column/comics-disassembled-ten-things-of-note-from-the-past-week-in-comics-led-by-boy-wonders/false-knees/ https://sktchd.com/column/comics-disassembled-ten-things-of-...

cynicalpeace 2y ago

Do they know what a hot shower feels like? They can describe it. But do they actually know? Have they experienced it? This is my point. Nerds keep dismissing physicality and experience. If your argument is a brain in a jar will be generally intelligent, I think that's pretty clearly wrong.

HDThoreaun 2y ago
See the responses section https://en.wikipedia.org/wiki/Knowledge_argument https://en.wikipedia.org/wiki/Knowledge_argument This idea certainly has been long considered but I personally reject it.
- cynicalpeace 2y ago
  While interesting, this is a separate thought experiment with its own quirks. Sort of a strawman, since my argument is formulated differently and simply argues that AIs need to be more than brains in jars for them to be considered generally intelligent. And that the only reason we think AIs can just be brains in jars is because many of the people developing them consider themselves as simply brains in jars.
  - HDThoreaun 2y ago
    Not really. The point of it is considering whether physical experience creates knowledge that is impossible to get otherwise. Thats the argument you are making no? If Mary learns nothing new when seeing red for the first time an AI would also learn nothing new when seeing red for the first time. > Do they know what a hot shower feels like? They can describe it. But do they actually know? Have they experienced it Is directly a knowledge argument
    - cynicalpeace 2y ago
      Mary in that thought experiment is not an LLM that has learned via text. She's acquired "all the physical information there is to obtain about what goes on when we see ripe tomatoes". This does not actually describe modern LLMs. It actually better describes a robot that has transcribed the location, temperature, and velocity of water drops from a hot shower to its memory. Again, this thought experiment has its own quirks. Also, it is an argument against physicalism, which I have no interest in debating. While it's tangentially related, my point is not for/against physicalism. My argument is about modern AI and it's ability to learn. If we put touch sensors, eyes, nose, a mechanism to collect physical data (legs) and even sex organs on an AI system, then it is more generally intelligent than before. It will have learned in a better fashion what a hot shower feels like and will be smarter for it.

magpi3 2y ago

Could it tell the difference between a dishwasher and a picture of a dishwasher on a wall? Or one painted onto a wall? Or a toy dishwasher? There is an essential idea of what makes something a dishwasher that LLM's will never be able to grasp no matter how many models you throw at them. They would have to fundamentally understand that what they are "seeing" is an electronic appliance connected to the plumbing that washes dishes. The sound of a running dishwasher, the heat you feel when you open one, and the wet, clean dishes is also part of that understanding.

viraptor 2y ago
Yes, it can tell a difference, up to the point where the boundaries are getting fuzzy. But the same thing applies to us all. Can you tell this is a dishwasher? https://www.amazon.com.au/Countertop-Dishwasher-Automatic-Vegetable-Apartments/dp/B0CLKTCVFC https://www.amazon.com.au/Countertop-Dishwasher-Automatic-Ve... Can you tell this is a drawing of a glass? https://www.deviantart.com/januarysnow13/art/Wine-Glass-Hyperrealistic-Drawing-506809946 https://www.deviantart.com/januarysnow13/art/Wine-Glass-Hype... Can you tell this is a toy? https://www.amazon.com.au/Theo-Klein-Miele-Washing-Machine/dp/B00UCK33HU https://www.amazon.com.au/Theo-Klein-Miele-Washing-Machine/d...
- magpi3 2y ago
  If I am limited to looking at pictures, then I am at the same disadvantage as the LLM, sure. The point is that people can experience and understand objects from a multitude of perspectives, both with our senses and the mental models we utilize to understand the object. Can LLMs do the same?
  - viraptor 2y ago
    That's not a disadvantage of LLM. You can start sending images from a camera moving around and you'll get many views as well. The capabilities here are the same as the eye-brain system - it can't move independently either.
    - 2y ago