4 ms·

Could GPT-4V be used for robotic applications? I am a bit confused here. It produces a text from an image, but how much actual understanding does it have? Can t

Could GPT-4V be used for robotic applications? I am a bit confused here. It produces a text from an image, but how much actual understanding does it have? Can the output somehow be used to do image segmentation and object detection and tracking?

voxic11 3y ago
There is a lot happening with multimodal models and robotics right now. PaLM-E is similar to GPT-4V in that its a LLM that has been trained on additional visual and robotics "sense" data. https://blog.research.google/2023/03/palm-e-embodied-multimodal-language.html https://blog.research.google/2023/03/palm-e-embodied-multimo...
kjander79 3y ago
> but how much actual understanding does it have? That's always the question, isn't it? The article does a pretty convincing job of showing that at least in the given examples, it has pretty good "understanding" of what's taking place in the scenes and what makes them remarkable to people, and 7 years for comparison is going back a long ways, just the last 2 or 3 years has been where much of the most interesting progress has revealed itself. Image segmentation, object detection and tracking, are all on display already here.
- RetroTechie 3y ago
  Just speculating on how the "understanding" may come about: In the given images above, it may be clear from context (text? tags? exif info etc?) of images in its training data, that it's unusual for people to be dragged on a rope behind a horse, very unusual / dangerous for 747 sized airplanes to fly on their side, or houses to be lying on their side on a beach. And hence, describe such a view with "unusual", "dramatic", etc. Would it even need to understand a conceptual meaning of those words? Apply label, done. Don't people work the same, in a way? Over the years we'll rarely see a house burning in person. We see news reports of such events mentioning people dead or severely burned. So after a while that 'training set' is enough to say: "person stumbling out of a burning house = something bad happened". Yes, humans may then reflect on how they would feel if placed in unlucky person's shoes, and rush to alleviate that person's pain. Or cringe by the thought of it. But in the end: maybe, just maybe, what human brains do isn't so special after all? Just training data, pattern match with external input & use results to self-reflect. (that last step not -yet- covered by GPT & co)
tonmoy 3y ago
You can ask GPT-4 for bounding box of each object in json format and it’ll give it to you. Don’t know if there is understanding, but it is definitely useful
- KRAKRISMOTT 3y ago
  How does it work? Does the encoder specifically pass the bounding box coordinates to the rest of the network as part of the embeddings?
  - Hidios 3y ago
    We don't understand how they work. We know how to make them, but much like the human brain, the very low level workings are beyond our understanding. Broadly it can be said that LLMs work by reducing uncertainty. Interestingly, human consciousness is also theorized by some to work the same way, react to the input in ways to reduce uncertainty.
  - tonmoy 3y ago
    Internally that is probably something similar that is happening, but as far as I can tell it was never designed to specifically do that. Here is an example of the things GPT-4V is capable of: https://blog.roboflow.com/gpt-4-vision/amp/ https://blog.roboflow.com/gpt-4-vision/amp/
moffkalast 3y ago
It's a bit slow for now, but yes end-to-end LMM systems will surely replace all classic general purpose robotic approaches eventually.
polygamous_bat 3y ago
> Could GPT-4V be used for robotic applications? I have a quote I like in this context, “Could you learn gymnastics just from watching videos?” However intelligent an internet trained model may be, I strongly believe you have to have some interaction with the real world to learn more complicated actions. So far, Pick and place, is all that’s been shown with the bigger models, so my hypothesis seems to be holding for now.
- famouswaffles 3y ago
  Pick and place isn't all you can do by any stretch. https://general-pattern-machines.github.io/ https://general-pattern-machines.github.io/ https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/ https://wayve.ai/thinking/lingo-natural-language-autonomous-... It's just the straightforward application. >I have a quote I like in this context, “Could you learn gymnastics just from watching videos?” It's not like language models learn by some sort of magic osmosis. They're not just "reading" text or "watching" images. They learn by predicting, failing and adjusting neurons based on the data. Text is their world and they are interacting with it.
- deleted 3y ago
  [deleted]

voxic11 3y ago

There is a lot happening with multimodal models and robotics right now. PaLM-E is similar to GPT-4V in that its a LLM that has been trained on additional visual and robotics "sense" data. https://blog.research.google/2023/03/palm-e-embodied-multimodal-language.html https://blog.research.google/2023/03/palm-e-embodied-multimo...

kjander79 3y ago

> but how much actual understanding does it have? That's always the question, isn't it? The article does a pretty convincing job of showing that at least in the given examples, it has pretty good "understanding" of what's taking place in the scenes and what makes them remarkable to people, and 7 years for comparison is going back a long ways, just the last 2 or 3 years has been where much of the most interesting progress has revealed itself. Image segmentation, object detection and tracking, are all on display already here.

RetroTechie 3y ago
Just speculating on how the "understanding" may come about: In the given images above, it may be clear from context (text? tags? exif info etc?) of images in its training data, that it's unusual for people to be dragged on a rope behind a horse, very unusual / dangerous for 747 sized airplanes to fly on their side, or houses to be lying on their side on a beach. And hence, describe such a view with "unusual", "dramatic", etc. Would it even need to understand a conceptual meaning of those words? Apply label, done. Don't people work the same, in a way? Over the years we'll rarely see a house burning in person. We see news reports of such events mentioning people dead or severely burned. So after a while that 'training set' is enough to say: "person stumbling out of a burning house = something bad happened". Yes, humans may then reflect on how they would feel if placed in unlucky person's shoes, and rush to alleviate that person's pain. Or cringe by the thought of it. But in the end: maybe, just maybe, what human brains do isn't so special after all? Just training data, pattern match with external input & use results to self-reflect. (that last step not -yet- covered by GPT & co)

tonmoy 3y ago

You can ask GPT-4 for bounding box of each object in json format and it’ll give it to you. Don’t know if there is understanding, but it is definitely useful

KRAKRISMOTT 3y ago
How does it work? Does the encoder specifically pass the bounding box coordinates to the rest of the network as part of the embeddings?
- Hidios 3y ago
  We don't understand how they work. We know how to make them, but much like the human brain, the very low level workings are beyond our understanding. Broadly it can be said that LLMs work by reducing uncertainty. Interestingly, human consciousness is also theorized by some to work the same way, react to the input in ways to reduce uncertainty.
- tonmoy 3y ago
  Internally that is probably something similar that is happening, but as far as I can tell it was never designed to specifically do that. Here is an example of the things GPT-4V is capable of: https://blog.roboflow.com/gpt-4-vision/amp/ https://blog.roboflow.com/gpt-4-vision/amp/

moffkalast 3y ago

It's a bit slow for now, but yes end-to-end LMM systems will surely replace all classic general purpose robotic approaches eventually.

polygamous_bat 3y ago

> Could GPT-4V be used for robotic applications? I have a quote I like in this context, “Could you learn gymnastics just from watching videos?” However intelligent an internet trained model may be, I strongly believe you have to have some interaction with the real world to learn more complicated actions. So far, Pick and place, is all that’s been shown with the bigger models, so my hypothesis seems to be holding for now.

famouswaffles 3y ago
Pick and place isn't all you can do by any stretch. https://general-pattern-machines.github.io/ https://general-pattern-machines.github.io/ https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/ https://wayve.ai/thinking/lingo-natural-language-autonomous-... It's just the straightforward application. >I have a quote I like in this context, “Could you learn gymnastics just from watching videos?” It's not like language models learn by some sort of magic osmosis. They're not just "reading" text or "watching" images. They learn by predicting, failing and adjusting neurons based on the data. Text is their world and they are interacting with it.
deleted 3y ago
[deleted]