5 ms·

Doesn't quite pass the Obama scale test[1] yet [1] http://karpathy.github.io/2012/10/22/state-of-computer-vision/ http://karpathy.github.io/2012/10/22/state-of

by danem 3y ago

Doesn't quite pass the Obama scale test[1] yet [1] http://karpathy.github.io/2012/10/22/state-of-computer-vision/ http://karpathy.github.io/2012/10/22/state-of-computer-visio... [2] https://imgur.com/LbgJvMQ https://imgur.com/LbgJvMQ

alpark3 3y ago
Some variations. https://imgur.com/a/nJGcwMO https://imgur.com/a/nJGcwMO I tried giving it more hints, but didn't seem to do anything useful.
- nextaccountic 3y ago
  It came close in both tests, but didn't recognize that Obama was tipping the scale To be honest I didn't either, and only noticed it after reading the 2012 Karpathy challenge > (...) > You recognize that Obama has his foot positioned just slightly on top of the scale. Notice the language I’m using: It is in terms of the 3D structure of the scene, not the position of the leg in the 2D coordinate system of the image. > You know how physics works: Obama is leaning in on the scale, which applies a force on it. Scale measures force that is applied on it, that’s how it works => it will over-estimate the weight of the person standing on it. > The person measuring his weight is not aware of Obama doing this. You derive this because you know his pose, you understand that the field of view of a person is finite, and you understand that he is not very likely to sense the slight push of Obama’s foot. > You understand that people are self-conscious about their weight. You also understand that he is reading off the scale measurement, and that shortly the over-estimated weight will confuse him because it will probably be much higher than what he expects. In other words, you reason about implications of the events that are about to unfold seconds after this photo was taken, and especially about the thoughts and how they will develop inside people’s heads. You also reason about what pieces of information are available to people. > There are people in the back who find the person’s imminent confusion funny. In other words you are reasoning about state of mind of people, and their view of the state of mind of another person. That’s getting frighteningly meta. > Finally, the fact that the perpetrator here is the president makes it maybe even a little more funnier. You understand what actions are more or less likely to be undertaken by different people based on their status and identity. ChatGPT got only the points above those (the "(...)")
pfd1986 3y ago
Can someone with access to gpt-4 multimodal test if it does?
- famouswaffles 3y ago
  It passed it. was already tested -https://twitter.com/karpathy/status/1635697741925064704 https://twitter.com/karpathy/status/1635697741925064704
  - CardenB 3y ago
    It’s not the Obama scale picture though? EDIT: I see now the link is different than the thumbnail in the post. Super impressive that GPT4 has solved this.
    - famouswaffles 3y ago
      read the op's question and karpathy's reply. ignore the example in the image.
  - bhaney 3y ago
    Personally I'd like to see the response it gave, not just a tweet saying it passed the test without showing the results of the test.
ekidd 3y ago
Nice example! LLaVA's language model appears to be weaker than GPT-4's. And we've already seen the featured GPT-4 demo where it explains a visual gag with a modified phone charging cable. So GPT 4 might get this. On the other hand, this test is almost certainly in GPT 4's training set. And I've seen GPT 4 correctly answer questions that appear on only a half dozen pages on the entire web. And since the Obama scale test is from 2012, there's a decent chance GPT 4 might just remember the answer.
- famouswaffles 3y ago
  Gpt-4 did pass it - https://twitter.com/karpathy/status/1635697741925064704 https://twitter.com/karpathy/status/1635697741925064704
  - meghan_rain 3y ago
    it didnt, the twitter post is popular and thus surely inadverently part of the training set
    - ljlolel 3y ago
      “Inadvertently”
ragebol 3y ago
Is that still a relevant test, if the blog post is used in the training data? How can I know it is or isn't included anyways?
- mnky9800n 3y ago
  I think you've highlighted the true issue. How do you know the model just doesn't have a glimpse of just about everything and looks that up when you ask about it?

by danem 3y ago

alpark3 3y ago

Some variations. https://imgur.com/a/nJGcwMO https://imgur.com/a/nJGcwMO I tried giving it more hints, but didn't seem to do anything useful.

nextaccountic 3y ago
It came close in both tests, but didn't recognize that Obama was tipping the scale To be honest I didn't either, and only noticed it after reading the 2012 Karpathy challenge > (...) > You recognize that Obama has his foot positioned just slightly on top of the scale. Notice the language I’m using: It is in terms of the 3D structure of the scene, not the position of the leg in the 2D coordinate system of the image. > You know how physics works: Obama is leaning in on the scale, which applies a force on it. Scale measures force that is applied on it, that’s how it works => it will over-estimate the weight of the person standing on it. > The person measuring his weight is not aware of Obama doing this. You derive this because you know his pose, you understand that the field of view of a person is finite, and you understand that he is not very likely to sense the slight push of Obama’s foot. > You understand that people are self-conscious about their weight. You also understand that he is reading off the scale measurement, and that shortly the over-estimated weight will confuse him because it will probably be much higher than what he expects. In other words, you reason about implications of the events that are about to unfold seconds after this photo was taken, and especially about the thoughts and how they will develop inside people’s heads. You also reason about what pieces of information are available to people. > There are people in the back who find the person’s imminent confusion funny. In other words you are reasoning about state of mind of people, and their view of the state of mind of another person. That’s getting frighteningly meta. > Finally, the fact that the perpetrator here is the president makes it maybe even a little more funnier. You understand what actions are more or less likely to be undertaken by different people based on their status and identity. ChatGPT got only the points above those (the "(...)")

pfd1986 3y ago

Can someone with access to gpt-4 multimodal test if it does?

famouswaffles 3y ago
It passed it. was already tested -https://twitter.com/karpathy/status/1635697741925064704 https://twitter.com/karpathy/status/1635697741925064704
- CardenB 3y ago
  It’s not the Obama scale picture though? EDIT: I see now the link is different than the thumbnail in the post. Super impressive that GPT4 has solved this.
  - famouswaffles 3y ago
    read the op's question and karpathy's reply. ignore the example in the image.
- bhaney 3y ago
  Personally I'd like to see the response it gave, not just a tweet saying it passed the test without showing the results of the test.

ekidd 3y ago

Nice example! LLaVA's language model appears to be weaker than GPT-4's. And we've already seen the featured GPT-4 demo where it explains a visual gag with a modified phone charging cable. So GPT 4 might get this. On the other hand, this test is almost certainly in GPT 4's training set. And I've seen GPT 4 correctly answer questions that appear on only a half dozen pages on the entire web. And since the Obama scale test is from 2012, there's a decent chance GPT 4 might just remember the answer.

famouswaffles 3y ago
Gpt-4 did pass it - https://twitter.com/karpathy/status/1635697741925064704 https://twitter.com/karpathy/status/1635697741925064704
- meghan_rain 3y ago
  it didnt, the twitter post is popular and thus surely inadverently part of the training set
  - ljlolel 3y ago
    “Inadvertently”

ragebol 3y ago

Is that still a relevant test, if the blog post is used in the training data? How can I know it is or isn't included anyways?

mnky9800n 3y ago
I think you've highlighted the true issue. How do you know the model just doesn't have a glimpse of just about everything and looks that up when you ask about it?