4 ms·

It's not too different from people. We also don't really "see" and mostly recall what we expect to see. What do you expect when the question is wrong "How many

It's not too different from people. We also don't really "see" and mostly recall what we expect to see. What do you expect when the question is wrong "How many legs does this animal have? Answer with a number" but it's not a picture of an animal. What are you supposed to do? Answer 0?

regularjack 1y ago
You answer "I don't know"
- amelius 1y ago
  What if that is not in your vocabulary?
ramoz 1y ago
This is interesting actually. And reminds me of something vaguely - a book or something that describes how human attention and the things we see are highly optimized by evolution. We often miss a lot of details in reality due to this.
- zehaeva 1y ago
  If it were a Fiction novel then might I suggest Blindsight by Peter Watts?
  - ramoz 1y ago
    not fiction. Maybe like a System 1 vs System 2 thing from Thinking, Fast and Slow by Kahneman. ChatGPT mentioned The Case Against Reality but I never read that, the idea was similar.
    - zehaeva 1y ago
      Maybe one of the books by Douglas Hofstadter? Godel Escher Bach? or I Am A Strange Loop?
vunderba 1y ago
That wasn't one of the questions - any reasonable person would have classified that chicken as an animal, albeit a mutant one. I would also hardly count many of these questions as "tricks" either. Take the chess example. A lot of my friends and myself have been playing chess since we were young children and we all know that a fully populated chess board has 32 pieces (heavily weighted in our internal training data), but not a single one of us would have gotten that question wrong.
- gowld 1y ago
  Don't be too literal. Imagine walking to a room an seeing someone grab a handful of chess pieces off of a set-up board, and proceed to fill bags with 4 pieces each. As they fill the 8th bag, they notice only 3 pieces are left. Are you confident that you would respond "I saw the board only had 31 pieces on it when you started", or might you reply "perhaos you dropped a piece on the floor"?
  - vunderba 1y ago
    I'm not. I'm referencing the paper - not some hypothetical abstract word problem. Imagine walking into a room, where the pieces are slowly morphing from staid Staunton structures into amorphous blobs of lava lamp Cthulhu nightmares. If a locomotive steam train from Denver passes within 15 meters of the room, how many passengers paid for the tickets using a cashier's check? Nobody's arguing that humans never take logical shortcuts or that those shortcuts can cause us to make errors. Some of the rebuttals in this thread are ridiculous. Like what if I forced you to stare at the surface of the sun followed by waterboarding for several hours, and then asked you to look at a 1000 different chess boards. Are you sure you wouldn't make a mistake? In the paper the various VLLMs are asked to double-check which still didn't make a difference. The argument is more along the lines that VLLMs (and multimodal LLMs) aren't really thinking in the same way that humans do. And if you REALLY need an example albeit a bit tangential - try this one out. Ask any SOTA (multimodal or otherwise) model such as gpt-image-1, Kontext, Imagen4, etc. for a five-leaf cover. It'll get it about 50% of the time. Now go and ask any kindergartener for the same thing.
wat10000 1y ago
Depending on the situation, I'd either walk away, or respond with, "What animal?"
enragedcacti 1y ago
Its true that our brains take lots of shortcuts when processing visual information but they don't necessarily parallel the shortcuts VLMs take. Humans are often very good at identifying anomalous instances of things they've seen thousands of times. No one has to tell you to look closely when you look at your partner in a mirror, you'll recognize it as 'off' immediately. Same for uncanny CGI of all types of things. If we were as sloppy as these models then VFX would be a hell of a lot easier. Ironically I think a lot of people in this thread are remembering things they learned about the faultiness of humans' visual memory and applying it to visual processing.

regularjack 1y ago

You answer "I don't know"

amelius 1y ago
What if that is not in your vocabulary?

ramoz 1y ago

This is interesting actually. And reminds me of something vaguely - a book or something that describes how human attention and the things we see are highly optimized by evolution. We often miss a lot of details in reality due to this.

zehaeva 1y ago
If it were a Fiction novel then might I suggest Blindsight by Peter Watts?
- ramoz 1y ago
  not fiction. Maybe like a System 1 vs System 2 thing from Thinking, Fast and Slow by Kahneman. ChatGPT mentioned The Case Against Reality but I never read that, the idea was similar.
  - zehaeva 1y ago
    Maybe one of the books by Douglas Hofstadter? Godel Escher Bach? or I Am A Strange Loop?

vunderba 1y ago

That wasn't one of the questions - any reasonable person would have classified that chicken as an animal, albeit a mutant one. I would also hardly count many of these questions as "tricks" either. Take the chess example. A lot of my friends and myself have been playing chess since we were young children and we all know that a fully populated chess board has 32 pieces (heavily weighted in our internal training data), but not a single one of us would have gotten that question wrong.

gowld 1y ago
Don't be too literal. Imagine walking to a room an seeing someone grab a handful of chess pieces off of a set-up board, and proceed to fill bags with 4 pieces each. As they fill the 8th bag, they notice only 3 pieces are left. Are you confident that you would respond "I saw the board only had 31 pieces on it when you started", or might you reply "perhaos you dropped a piece on the floor"?
- vunderba 1y ago
  I'm not. I'm referencing the paper - not some hypothetical abstract word problem. Imagine walking into a room, where the pieces are slowly morphing from staid Staunton structures into amorphous blobs of lava lamp Cthulhu nightmares. If a locomotive steam train from Denver passes within 15 meters of the room, how many passengers paid for the tickets using a cashier's check? Nobody's arguing that humans never take logical shortcuts or that those shortcuts can cause us to make errors. Some of the rebuttals in this thread are ridiculous. Like what if I forced you to stare at the surface of the sun followed by waterboarding for several hours, and then asked you to look at a 1000 different chess boards. Are you sure you wouldn't make a mistake? In the paper the various VLLMs are asked to double-check which still didn't make a difference. The argument is more along the lines that VLLMs (and multimodal LLMs) aren't really thinking in the same way that humans do. And if you REALLY need an example albeit a bit tangential - try this one out. Ask any SOTA (multimodal or otherwise) model such as gpt-image-1, Kontext, Imagen4, etc. for a five-leaf cover. It'll get it about 50% of the time. Now go and ask any kindergartener for the same thing.

wat10000 1y ago

Depending on the situation, I'd either walk away, or respond with, "What animal?"

enragedcacti 1y ago

Its true that our brains take lots of shortcuts when processing visual information but they don't necessarily parallel the shortcuts VLMs take. Humans are often very good at identifying anomalous instances of things they've seen thousands of times. No one has to tell you to look closely when you look at your partner in a mirror, you'll recognize it as 'off' immediately. Same for uncanny CGI of all types of things. If we were as sloppy as these models then VFX would be a hell of a lot easier. Ironically I think a lot of people in this thread are remembering things they learned about the faultiness of humans' visual memory and applying it to visual processing.