4 ms·

In my experience GPT4/V is pretty bad at those specifically, not necessarily around abstraction in general. The problem with a statement like is that it leaves

In my experience GPT4/V is pretty bad at those specifically, not necessarily around abstraction in general. The problem with a statement like is that it leaves the door open to accepting any kind of canned generality as "abstraction in general". Abstract reasoning is indeed a fuzzy/slippery concept and spatial reason may not capture it well but I'm pretty sure it captures it better a general impression of ChatGPT. ...since it has no body, no world, no space; it "lives" in the realm of text. There's a bizarre anthropomorphism on this thread, both reflexively compare this software system to a blind human and the implicit call to be considerate of this thing's supposed disability.

famouswaffles 3y ago
>The problem with a statement like is that it leaves the door open to accepting any kind of canned generality as "abstraction in general". Not really https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196
- joe_the_user 3y ago
  Nah, By the very fact that there's paper here, whatever it's merit, the authors of the paper have codified their concept of generality and this doesn't validate the point I was replying to, which was essentially "my impression/feeling" is that it is better".
  - famouswaffles 3y ago
    Point is that it's good at abstract reasoning that isn't spatially grounded like in that paper. So it's not really leaving any door open. It's not a cop out. That's just how it is.
int_19h 3y ago
Why is it bizarre to consider the limitations inherent in the input data on which the model is trained? Fundamentally, it still "sees" the world through text, and the extent to which it can "understand" spatial relationships is defined by that. It seems utterly unsurprising that this leads to very poor grasp of the actual concepts behind what things like "above" or "left" are - the text that humans produce when talking about such things kinda relies on the reader having their own experience (if not vision, then at least body awareness) that can be mapped to those concepts. You can explain "left" and "right" to a human by telling them which of their hands is which, and I can't help but wonder what the actual information payload this constitutes when you consider the body spatial awareness that is brought into context by association.
- dragonwriter 3y ago
  > Fundamentally, it still "sees" the world through text Fundamentally, it "sees the world" [0] through tokens, which are not text. [0] Also a bad metaphor, but...
  - int_19h 3y ago
    Technically true, but when those tokens are 1:1 mapped to text, I think we can simplify this down without losing anything important. Of course, once you start using tokens for other things - as multimodal LMs already do - that changes. But this current crop of model still has visual modality in its infancy IMO, and gauging the overall performance of model as a whole based on that is very questionable.
    - dragonwriter 3y ago
      > Technically true, but when those tokens are 1:1 mapped to text I don't know what GPT-4V does particularly, but my understanding is that multimodal models very often have an expanded token space with special tokens related to image handling, so, literally, there is not a 1:1 relationship of tokens to text.
  - Jensson 3y ago
    A string of tokens is text. Tokens is just another alphabet, like Japanese letters having many representations for the same sounds and a letter can be entire words sometimes.
- joe_the_user 3y ago
  Why is it bizarre to consider the limitations inherent in the input data on which the model is trained? Sure the thing is limited, the study is demonstration of this (and general purpose abilities have been claimed for LLMs at various point). I was pushing back against the "it's like a blind person" anthropomorphizing argument [edit: especially the assumption these things learn through experience and reflection which the parent also makes]. Maybe if the thing "had eyes", it could learn spatial information and maybe it couldn't (though it would take a lot of work to make that metaphor meaningful). The thing certainly doesn't learn text in the fashion that human learns speech since humans don't digest the entire Internet before they can speak.

famouswaffles 3y ago

>The problem with a statement like is that it leaves the door open to accepting any kind of canned generality as "abstraction in general". Not really https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196

joe_the_user 3y ago
Nah, By the very fact that there's paper here, whatever it's merit, the authors of the paper have codified their concept of generality and this doesn't validate the point I was replying to, which was essentially "my impression/feeling" is that it is better".
- famouswaffles 3y ago
  Point is that it's good at abstract reasoning that isn't spatially grounded like in that paper. So it's not really leaving any door open. It's not a cop out. That's just how it is.

int_19h 3y ago

Why is it bizarre to consider the limitations inherent in the input data on which the model is trained? Fundamentally, it still "sees" the world through text, and the extent to which it can "understand" spatial relationships is defined by that. It seems utterly unsurprising that this leads to very poor grasp of the actual concepts behind what things like "above" or "left" are - the text that humans produce when talking about such things kinda relies on the reader having their own experience (if not vision, then at least body awareness) that can be mapped to those concepts. You can explain "left" and "right" to a human by telling them which of their hands is which, and I can't help but wonder what the actual information payload this constitutes when you consider the body spatial awareness that is brought into context by association.

dragonwriter 3y ago
> Fundamentally, it still "sees" the world through text Fundamentally, it "sees the world" [0] through tokens, which are not text. [0] Also a bad metaphor, but...
- int_19h 3y ago
  Technically true, but when those tokens are 1:1 mapped to text, I think we can simplify this down without losing anything important. Of course, once you start using tokens for other things - as multimodal LMs already do - that changes. But this current crop of model still has visual modality in its infancy IMO, and gauging the overall performance of model as a whole based on that is very questionable.
  - dragonwriter 3y ago
    > Technically true, but when those tokens are 1:1 mapped to text I don't know what GPT-4V does particularly, but my understanding is that multimodal models very often have an expanded token space with special tokens related to image handling, so, literally, there is not a 1:1 relationship of tokens to text.
- Jensson 3y ago
  A string of tokens is text. Tokens is just another alphabet, like Japanese letters having many representations for the same sounds and a letter can be entire words sometimes.
joe_the_user 3y ago
Why is it bizarre to consider the limitations inherent in the input data on which the model is trained? Sure the thing is limited, the study is demonstration of this (and general purpose abilities have been claimed for LLMs at various point). I was pushing back against the "it's like a blind person" anthropomorphizing argument [edit: especially the assumption these things learn through experience and reflection which the parent also makes]. Maybe if the thing "had eyes", it could learn spatial information and maybe it couldn't (though it would take a lot of work to make that metaphor meaningful). The thing certainly doesn't learn text in the fashion that human learns speech since humans don't digest the entire Internet before they can speak.