6 ms·

"Building Machines That Learn and Think Like People", 7 Years Later

106 points by che_shr_cat 3y ago

mustafa_pasi 3y ago
Could GPT-4V be used for robotic applications? I am a bit confused here. It produces a text from an image, but how much actual understanding does it have? Can the output somehow be used to do image segmentation and object detection and tracking?
- voxic11 3y ago
  There is a lot happening with multimodal models and robotics right now. PaLM-E is similar to GPT-4V in that its a LLM that has been trained on additional visual and robotics "sense" data. https://blog.research.google/2023/03/palm-e-embodied-multimodal-language.html https://blog.research.google/2023/03/palm-e-embodied-multimo...
- kjander79 3y ago
  > but how much actual understanding does it have? That's always the question, isn't it? The article does a pretty convincing job of showing that at least in the given examples, it has pretty good "understanding" of what's taking place in the scenes and what makes them remarkable to people, and 7 years for comparison is going back a long ways, just the last 2 or 3 years has been where much of the most interesting progress has revealed itself. Image segmentation, object detection and tracking, are all on display already here.
  - RetroTechie 3y ago
    Just speculating on how the "understanding" may come about: In the given images above, it may be clear from context (text? tags? exif info etc?) of images in its training data, that it's unusual for people to be dragged on a rope behind a horse, very unusual / dangerous for 747 sized airplanes to fly on their side, or houses to be lying on their side on a beach. And hence, describe such a view with "unusual", "dramatic", etc. Would it even need to understand a conceptual meaning of those words? Apply label, done. Don't people work the same, in a way? Over the years we'll rarely see a house burning in person. We see news reports of such events mentioning people dead or severely burned. So after a while that 'training set' is enough to say: "person stumbling out of a burning house = something bad happened". Yes, humans may then reflect on how they would feel if placed in unlucky person's shoes, and rush to alleviate that person's pain. Or cringe by the thought of it. But in the end: maybe, just maybe, what human brains do isn't so special after all? Just training data, pattern match with external input & use results to self-reflect. (that last step not -yet- covered by GPT & co)
- tonmoy 3y ago
  You can ask GPT-4 for bounding box of each object in json format and it’ll give it to you. Don’t know if there is understanding, but it is definitely useful
  - KRAKRISMOTT 3y ago
    How does it work? Does the encoder specifically pass the bounding box coordinates to the rest of the network as part of the embeddings?
    - Hidios 3y ago
      We don't understand how they work. We know how to make them, but much like the human brain, the very low level workings are beyond our understanding. Broadly it can be said that LLMs work by reducing uncertainty. Interestingly, human consciousness is also theorized by some to work the same way, react to the input in ways to reduce uncertainty.
    - tonmoy 3y ago
      Internally that is probably something similar that is happening, but as far as I can tell it was never designed to specifically do that. Here is an example of the things GPT-4V is capable of: https://blog.roboflow.com/gpt-4-vision/amp/ https://blog.roboflow.com/gpt-4-vision/amp/
- moffkalast 3y ago
  It's a bit slow for now, but yes end-to-end LMM systems will surely replace all classic general purpose robotic approaches eventually.
- polygamous_bat 3y ago
  > Could GPT-4V be used for robotic applications? I have a quote I like in this context, “Could you learn gymnastics just from watching videos?” However intelligent an internet trained model may be, I strongly believe you have to have some interaction with the real world to learn more complicated actions. So far, Pick and place, is all that’s been shown with the bigger models, so my hypothesis seems to be holding for now.
  - famouswaffles 3y ago
    Pick and place isn't all you can do by any stretch. https://general-pattern-machines.github.io/ https://general-pattern-machines.github.io/ https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/ https://wayve.ai/thinking/lingo-natural-language-autonomous-... It's just the straightforward application. >I have a quote I like in this context, “Could you learn gymnastics just from watching videos?” It's not like language models learn by some sort of magic osmosis. They're not just "reading" text or "watching" images. They learn by predicting, failing and adjusting neurons based on the data. Text is their world and they are interacting with it.
  - deleted 3y ago
    [deleted]
sgt101 3y ago
Do you think that these images were in the GPT4 training set? maybe...
- sgt101 3y ago
  honestly, what sort of internal model of ML and AI do people who write blogs lik e this have?
- kjander79 3y ago
  Actually that's an interesting point, since the images are already in use in the article this one is responding to. It seems likely.
Animats 3y ago
Indeed, there has been much progress. The next big fundamental problem is "hallucination", or being totally wrong without detecting it.
- konschubert 3y ago
  Same problem with human brains.
- greatpostman 3y ago
  The hallucinations are what allow for the intelligence. It’s a feature not a bug
M4v3R 3y ago
GPT-4V is mind blowing, it surprising to me that it gets so little attention here on HN, because after playing around with it I get the same sense of excitement I got when I tried the original ChatGPT. The level of understanding of what is going on in an image is leagues ahead of what we had until this point, ahead of Bard and basically everything else I already saw. I tested it with a bunch of photos I made and images it could not have seen in its training data, and most of the time it nailed them perfectly. Its OCR capabilities are top notch, but this is combined with a spatial understanding of how text relates to other parts of the image. It can take a photo of a wall monthly calendar with hand scribbles on it and give you a list of events for each day. It can guess where a specific photo was taken just by analysing the elements present on the photo like the foliage, architecture, car license plates, etc (without being specifically prompted to do so). It can correctly identify multiple plants from a same photo. Gave it a photo of a Montessori set for teaching math (some wooden blocks with numbers and dots on them, no branding on them) and it guessed exactly what it was. And all of that just from two days of testing. Here are just few examples: [1] https://i.imgur.com/cV3dVOf.png https://i.imgur.com/cV3dVOf.png - Gave it a screenshot from Final Fantasy VII from a boss battle. It correctly identified the party members, and their stats even though the text and labels are a bit all over the place. [2] https://i.imgur.com/WeXhP7V.png https://i.imgur.com/WeXhP7V.png - A photo I shot on my vacation, that didn't really contain any major landmarks, and yet it still somehow figured out from the architecture (and house colors) the exact location of it. I tried this game with several photos and it is very good at it, far better than I could ever be if I saw these photos for the first time. [3] https://i.imgur.com/HgwYv6q.png https://i.imgur.com/HgwYv6q.png - A screenshot of a worksheet from the Human Shader Project. I just asked it to solve it for given X/Y values and it did, its answer was 100% correct (here's the second part of its answer: https://i.imgur.com/RZF2r7v.png https://i.imgur.com/RZF2r7v.png) [4] https://i.imgur.com/12xg4qU.png https://i.imgur.com/12xg4qU.png - A photo of a highly reflective microwave inside a shopping mall. This was given to my by a friend who shot this personally and to be honest I didn't catch at first that this is a microwave, and yet GPT-4V figured that out. [5] https://i.imgur.com/qSifni5.png https://i.imgur.com/qSifni5.png - a good old fashioned "find the path connecting one object to the other" puzzle. Correctly identified the right path (this one was taken from the internet so there is a slight chance it saw it in the training data and somehow got the solution for it from the accompanying text, although I couldn't find any instance of it). Edit: To confirm that [5] was not a fluke I hand drew my own version of this puzzle, took a picture and uploaded it, GPT-4V nailed this one too: https://i.imgur.com/8NgWhzw.png https://i.imgur.com/8NgWhzw.png
two_in_one 3y ago
I played with it, a cool thing: It can write a poem from image. It can read text from image and 'understand' it. Or even specific part of the text. You can say "look at the bottom line". It can recognize and list songs from album cover. It recognizes famous paintings. Even if only a fragment is given. It can be used to create image-text datasets for generative and recognition tasks. Not sure how much this would cost.
- golergka 3y ago
  I was at a restoraunt today and was curious what the sauce was made of. I took a photo of the dish and given the city I was in, gpt4 identified the name of the dish and all ingredients correctly, as I verified with the waiter afterwards. Felt like magic.
  - endofreach 3y ago
    Did you strip the exif or disabled gps locations on photos?
    - golergka 3y ago
      No, I didn't. Why would I bother? I explicitly told ChatGPT what city I was in.
drewcoo 3y ago
So it draws pictures like a grade school boy, a human whose "training set" involves TV and movies. For seven years that doesn't seem unreasonable.
jccalhoun 3y ago
As a college professor the descriptions from ChatGPT remind me a lot of freshmen writing: they are often flailing around on extraneous details and have difficulty determining what is and isn't important. It will be interesting to see how it improves.
- BoorishBears 3y ago
  GPT-4 in the web UI defaults to trying to describe the image very tentatively and in general, and it seems intentional: doing that improves performance if you're expecting follow up questions similarly to chain of thought. I expect the API with low temperature will be much more decisive in describing images
scotty79 3y ago
I was terrible at this task as a kid. When asked to describe an image I usually volunteered single piece of information about it and I had to be prompted multiple times and asked leading questions to observe and describe more of it. GPT-4V beats me even now. There were a lot of information in the descriptions I wouldn't notice or include without being specifically asked about them.

106 points by che_shr_cat 3y ago

Could GPT-4V be used for robotic applications? I am a bit confused here. It produces a text from an image, but how much actual understanding does it have? Can the output somehow be used to do image segmentation and object detection and tracking?

voxic11 3y ago
There is a lot happening with multimodal models and robotics right now. PaLM-E is similar to GPT-4V in that its a LLM that has been trained on additional visual and robotics "sense" data. https://blog.research.google/2023/03/palm-e-embodied-multimodal-language.html https://blog.research.google/2023/03/palm-e-embodied-multimo...
kjander79 3y ago
> but how much actual understanding does it have? That's always the question, isn't it? The article does a pretty convincing job of showing that at least in the given examples, it has pretty good "understanding" of what's taking place in the scenes and what makes them remarkable to people, and 7 years for comparison is going back a long ways, just the last 2 or 3 years has been where much of the most interesting progress has revealed itself. Image segmentation, object detection and tracking, are all on display already here.
- RetroTechie 3y ago
  Just speculating on how the "understanding" may come about: In the given images above, it may be clear from context (text? tags? exif info etc?) of images in its training data, that it's unusual for people to be dragged on a rope behind a horse, very unusual / dangerous for 747 sized airplanes to fly on their side, or houses to be lying on their side on a beach. And hence, describe such a view with "unusual", "dramatic", etc. Would it even need to understand a conceptual meaning of those words? Apply label, done. Don't people work the same, in a way? Over the years we'll rarely see a house burning in person. We see news reports of such events mentioning people dead or severely burned. So after a while that 'training set' is enough to say: "person stumbling out of a burning house = something bad happened". Yes, humans may then reflect on how they would feel if placed in unlucky person's shoes, and rush to alleviate that person's pain. Or cringe by the thought of it. But in the end: maybe, just maybe, what human brains do isn't so special after all? Just training data, pattern match with external input & use results to self-reflect. (that last step not -yet- covered by GPT & co)
tonmoy 3y ago
You can ask GPT-4 for bounding box of each object in json format and it’ll give it to you. Don’t know if there is understanding, but it is definitely useful
- KRAKRISMOTT 3y ago
  How does it work? Does the encoder specifically pass the bounding box coordinates to the rest of the network as part of the embeddings?
  - Hidios 3y ago
    We don't understand how they work. We know how to make them, but much like the human brain, the very low level workings are beyond our understanding. Broadly it can be said that LLMs work by reducing uncertainty. Interestingly, human consciousness is also theorized by some to work the same way, react to the input in ways to reduce uncertainty.
  - tonmoy 3y ago
    Internally that is probably something similar that is happening, but as far as I can tell it was never designed to specifically do that. Here is an example of the things GPT-4V is capable of: https://blog.roboflow.com/gpt-4-vision/amp/ https://blog.roboflow.com/gpt-4-vision/amp/
moffkalast 3y ago
It's a bit slow for now, but yes end-to-end LMM systems will surely replace all classic general purpose robotic approaches eventually.
polygamous_bat 3y ago
> Could GPT-4V be used for robotic applications? I have a quote I like in this context, “Could you learn gymnastics just from watching videos?” However intelligent an internet trained model may be, I strongly believe you have to have some interaction with the real world to learn more complicated actions. So far, Pick and place, is all that’s been shown with the bigger models, so my hypothesis seems to be holding for now.
- famouswaffles 3y ago
  Pick and place isn't all you can do by any stretch. https://general-pattern-machines.github.io/ https://general-pattern-machines.github.io/ https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/ https://wayve.ai/thinking/lingo-natural-language-autonomous-... It's just the straightforward application. >I have a quote I like in this context, “Could you learn gymnastics just from watching videos?” It's not like language models learn by some sort of magic osmosis. They're not just "reading" text or "watching" images. They learn by predicting, failing and adjusting neurons based on the data. Text is their world and they are interacting with it.
- deleted 3y ago
  [deleted]

sgt101 3y ago

Do you think that these images were in the GPT4 training set? maybe...

sgt101 3y ago
honestly, what sort of internal model of ML and AI do people who write blogs lik e this have?
kjander79 3y ago
Actually that's an interesting point, since the images are already in use in the article this one is responding to. It seems likely.

Animats 3y ago

Indeed, there has been much progress. The next big fundamental problem is "hallucination", or being totally wrong without detecting it.

konschubert 3y ago
Same problem with human brains.
greatpostman 3y ago
The hallucinations are what allow for the intelligence. It’s a feature not a bug

M4v3R 3y ago

GPT-4V is mind blowing, it surprising to me that it gets so little attention here on HN, because after playing around with it I get the same sense of excitement I got when I tried the original ChatGPT. The level of understanding of what is going on in an image is leagues ahead of what we had until this point, ahead of Bard and basically everything else I already saw. I tested it with a bunch of photos I made and images it could not have seen in its training data, and most of the time it nailed them perfectly. Its OCR capabilities are top notch, but this is combined with a spatial understanding of how text relates to other parts of the image. It can take a photo of a wall monthly calendar with hand scribbles on it and give you a list of events for each day. It can guess where a specific photo was taken just by analysing the elements present on the photo like the foliage, architecture, car license plates, etc (without being specifically prompted to do so). It can correctly identify multiple plants from a same photo. Gave it a photo of a Montessori set for teaching math (some wooden blocks with numbers and dots on them, no branding on them) and it guessed exactly what it was. And all of that just from two days of testing. Here are just few examples: [1] https://i.imgur.com/cV3dVOf.png https://i.imgur.com/cV3dVOf.png - Gave it a screenshot from Final Fantasy VII from a boss battle. It correctly identified the party members, and their stats even though the text and labels are a bit all over the place. [2] https://i.imgur.com/WeXhP7V.png https://i.imgur.com/WeXhP7V.png - A photo I shot on my vacation, that didn't really contain any major landmarks, and yet it still somehow figured out from the architecture (and house colors) the exact location of it. I tried this game with several photos and it is very good at it, far better than I could ever be if I saw these photos for the first time. [3] https://i.imgur.com/HgwYv6q.png https://i.imgur.com/HgwYv6q.png - A screenshot of a worksheet from the Human Shader Project. I just asked it to solve it for given X/Y values and it did, its answer was 100% correct (here's the second part of its answer: https://i.imgur.com/RZF2r7v.png https://i.imgur.com/RZF2r7v.png) [4] https://i.imgur.com/12xg4qU.png https://i.imgur.com/12xg4qU.png - A photo of a highly reflective microwave inside a shopping mall. This was given to my by a friend who shot this personally and to be honest I didn't catch at first that this is a microwave, and yet GPT-4V figured that out. [5] https://i.imgur.com/qSifni5.png https://i.imgur.com/qSifni5.png - a good old fashioned "find the path connecting one object to the other" puzzle. Correctly identified the right path (this one was taken from the internet so there is a slight chance it saw it in the training data and somehow got the solution for it from the accompanying text, although I couldn't find any instance of it). Edit: To confirm that [5] was not a fluke I hand drew my own version of this puzzle, took a picture and uploaded it, GPT-4V nailed this one too: https://i.imgur.com/8NgWhzw.png https://i.imgur.com/8NgWhzw.png

two_in_one 3y ago

I played with it, a cool thing: It can write a poem from image. It can read text from image and 'understand' it. Or even specific part of the text. You can say "look at the bottom line". It can recognize and list songs from album cover. It recognizes famous paintings. Even if only a fragment is given. It can be used to create image-text datasets for generative and recognition tasks. Not sure how much this would cost.

golergka 3y ago
I was at a restoraunt today and was curious what the sauce was made of. I took a photo of the dish and given the city I was in, gpt4 identified the name of the dish and all ingredients correctly, as I verified with the waiter afterwards. Felt like magic.
- endofreach 3y ago
  Did you strip the exif or disabled gps locations on photos?
  - golergka 3y ago
    No, I didn't. Why would I bother? I explicitly told ChatGPT what city I was in.

drewcoo 3y ago

So it draws pictures like a grade school boy, a human whose "training set" involves TV and movies. For seven years that doesn't seem unreasonable.

jccalhoun 3y ago

As a college professor the descriptions from ChatGPT remind me a lot of freshmen writing: they are often flailing around on extraneous details and have difficulty determining what is and isn't important. It will be interesting to see how it improves.

BoorishBears 3y ago
GPT-4 in the web UI defaults to trying to describe the image very tentatively and in general, and it seems intentional: doing that improves performance if you're expecting follow up questions similarly to chain of thought. I expect the API with low temperature will be much more decisive in describing images

scotty79 3y ago

I was terrible at this task as a kid. When asked to describe an image I usually volunteered single piece of information about it and I had to be prompted multiple times and asked leading questions to observe and describe more of it. GPT-4V beats me even now. There were a lot of information in the descriptions I wouldn't notice or include without being specifically asked about them.