3 ms·

But there's also plenty of things I haven't seen in real life and have only read about in books or text. Does it mean that I wouldn't know what they are as well

But there's also plenty of things I haven't seen in real life and have only read about in books or text. Does it mean that I wouldn't know what they are as well? In addition you can feed LLMs image, audio input as well, and in theory you could feed any sensory data that we get. In fact it can describe what is on an image, to quite a lot of detail. Is knowing what a dog is being able to recognize it on an image, and is knowing what a woof is being able to recognize it in an audio? Ultimately, after all, all of it can be represented as input tokens, binary data or otherwise. We also receive constant input from the World, which could be represented in any various different forms and processing layers can transform it into different helpful symbolic things to problem solve based on that. LLMs in order to be able to logically consider what a dog might do given certain situations, would have needed to build some form of neural representation of the idea to be able to produce the output it can, that goes beyond parroting the training data in my view. I can give it an input in some interesting configurations and permutations and it's able to logically use the concepts to figure out what might happen.

AndrewDucker 2y ago
"Does it mean that I wouldn't know what they are as well?" Are they explained in terms of things that you have seen in real life? Or in ways that build on that? Then you can certainly gain that by building on those models. And yes, if you can feed an attention-based neural network (not an LLM, those are specifically Language-based) sense data (and preferably action data too, for how it interacts with the world) then that would give it a picture of the world. And once it can recognise a dog then you can say "And we call this a dog!". There's research in to this already - see this piece https://www.economist.com/science-and-technology/2024/06/05/robots-are-suddenly-getting-cleverer-whats-changed https://www.economist.com/science-and-technology/2024/06/05/... (you may need to use archive.is to read the archive of that page). ==== These “vision-language-action models” (VLAMs) take in text and images, plus data relating to the robot’s presence in the physical world, including the readings on internal sensors, the degree of rotation of different joints and the positions of actuators (such as grippers, or the fingers of a robot’s hands). The resulting models can then answer questions about a scene, such as “can you see an apple?” But they can also predict how a robot arm needs to move to pick that apple up, as well as how this will affect what the world looks like." Grounding the model’s perception in the real world in this way greatly reduces hallucinations (the tendency for AI models to make things up and get things wrong). ====
- mewpmewp2 2y ago
  But GPT-4o for example as a multimodal model is already able to take images as an input, so does it have enough to know what a dog is in your opinion? Even though it's able to process images it's still being called an LLM, but multimodal LLM.

AndrewDucker 2y ago

"Does it mean that I wouldn't know what they are as well?" Are they explained in terms of things that you have seen in real life? Or in ways that build on that? Then you can certainly gain that by building on those models. And yes, if you can feed an attention-based neural network (not an LLM, those are specifically Language-based) sense data (and preferably action data too, for how it interacts with the world) then that would give it a picture of the world. And once it can recognise a dog then you can say "And we call this a dog!". There's research in to this already - see this piece https://www.economist.com/science-and-technology/2024/06/05/robots-are-suddenly-getting-cleverer-whats-changed https://www.economist.com/science-and-technology/2024/06/05/... (you may need to use archive.is to read the archive of that page). ==== These “vision-language-action models” (VLAMs) take in text and images, plus data relating to the robot’s presence in the physical world, including the readings on internal sensors, the degree of rotation of different joints and the positions of actuators (such as grippers, or the fingers of a robot’s hands). The resulting models can then answer questions about a scene, such as “can you see an apple?” But they can also predict how a robot arm needs to move to pick that apple up, as well as how this will affect what the world looks like." Grounding the model’s perception in the real world in this way greatly reduces hallucinations (the tendency for AI models to make things up and get things wrong). ====

mewpmewp2 2y ago
But GPT-4o for example as a multimodal model is already able to take images as an input, so does it have enough to know what a dog is in your opinion? Even though it's able to process images it's still being called an LLM, but multimodal LLM.