3 ms·

I’ve been imagining “reasoning” as meaning “referring to an abstraction”. The idea being if you wanted to write a script to tell you how far an arrow of a given

I’ve been imagining “reasoning” as meaning “referring to an abstraction”. The idea being if you wanted to write a script to tell you how far an arrow of a given weight will fly, you’ll probably use some physics simulation (or even just a coefficient to multiply the input weight by) to abstract the arrow in flight. Then you can answer questions around that abstraction. The alternative is having 1000 different results for different kinds of arrows, and averaging out the results for the ones similar to the input arrow. An LLM is working on text tokens, it’s trying to give the most statistically common next token based on everything that’s been fed into it. Does that statistical model abstract the objects and concepts it talks about? Eh? I don’t know

cjbprime 4y ago
This is one of those comments that begs for actually using LLMs. Yes, an advanced LLM like GPT-4 will absolutely show evidence of an abstracted world model, correctly answering questions about where person X thinks object Y is, or what relation two people have to each other, or playing a text adventure game while clearly following an internal map it has created for itself.
- graypegg 4y ago
  So I get that the evidence is that you can do insanely complex things with consistency, like playing a text adventure. You pick up a thing and it stays in your inventory etc, ergo, it’s modelled a character with an inventory in a text adventure. vvv But, coming at this as *respectfully and curiously* as I can here: ^^^ I guess I’m a little more skeptical about attributing magic to something that I know is only working from words as a source. Like if it’s seen NOUN really close to VERB a lot, it can “assume” (read: bias output) that NOUN will VERB in the context it sees it in, that’s an abstraction! But a weak one right? Language is pretty inaccurate. The fact that NLP was a whole booming research field a few years ago tells me that just parsing language consistently is a difficult problem. So, I know I can’t give you any convincing argument there since we’re already mentioning how people are giving untestable definitions of “reasoning” and “abstraction” in this thread. And im no better in that respect here, BUT I guess my best inkling of what feels “off” is a lack of precision, inherent to the medium in which it operates which makes it feel like a true abstraction is impossible in an LLM.
  - cjbprime 4y ago
    > something that I know is only working from words as a source. Your intuition about the architecture might be causing the distrust. Yes, the ends of the model are words in and words out (although GPT-4 is now multi-modal and can accept images as input too), but that doesn't mean the calculations in the middle are dealing in words. The "language" in "large language model" is misleading. It's a large generalized model that imports and exports language.
    - graypegg 4y ago
      That is totally possible, the explosion of stuff around this space has kind of blindsided me, so I don’t have much intuition for precisely how these models are built. It all still feels rather “attribute magic to complexity” to say that there’s some human element skimmed off the patterns of our language, hidden in the coefficients somewhere. I’m also not a magician, so really who am I to say :)
kenjackson 4y ago
Does it really provide the statistically most common next token? I don’t think that’s even true. And we’ve seen what looks like emergent behavior as parameters increase. I suspect it is actually creating abstractions at a deeper level than standard DNNs.
- graypegg 4y ago
  I’d be curious to know! I get skeptical when it starts moving into the “emergent properties of language” but language was built by fleshy neurons, it could be that some inherent pattern emerges in the language we speak that reflects the inner workings of our brain, which by trying to emulate makes a similar brain on the other end. However, that feels a bit magical to me at the moment. Still interesting to be along for the ride humanity is going on!

cjbprime 4y ago

This is one of those comments that begs for actually using LLMs. Yes, an advanced LLM like GPT-4 will absolutely show evidence of an abstracted world model, correctly answering questions about where person X thinks object Y is, or what relation two people have to each other, or playing a text adventure game while clearly following an internal map it has created for itself.

graypegg 4y ago
So I get that the evidence is that you can do insanely complex things with consistency, like playing a text adventure. You pick up a thing and it stays in your inventory etc, ergo, it’s modelled a character with an inventory in a text adventure. vvv But, coming at this as *respectfully and curiously* as I can here: ^^^ I guess I’m a little more skeptical about attributing magic to something that I know is only working from words as a source. Like if it’s seen NOUN really close to VERB a lot, it can “assume” (read: bias output) that NOUN will VERB in the context it sees it in, that’s an abstraction! But a weak one right? Language is pretty inaccurate. The fact that NLP was a whole booming research field a few years ago tells me that just parsing language consistently is a difficult problem. So, I know I can’t give you any convincing argument there since we’re already mentioning how people are giving untestable definitions of “reasoning” and “abstraction” in this thread. And im no better in that respect here, BUT I guess my best inkling of what feels “off” is a lack of precision, inherent to the medium in which it operates which makes it feel like a true abstraction is impossible in an LLM.
- cjbprime 4y ago
  > something that I know is only working from words as a source. Your intuition about the architecture might be causing the distrust. Yes, the ends of the model are words in and words out (although GPT-4 is now multi-modal and can accept images as input too), but that doesn't mean the calculations in the middle are dealing in words. The "language" in "large language model" is misleading. It's a large generalized model that imports and exports language.
  - graypegg 4y ago
    That is totally possible, the explosion of stuff around this space has kind of blindsided me, so I don’t have much intuition for precisely how these models are built. It all still feels rather “attribute magic to complexity” to say that there’s some human element skimmed off the patterns of our language, hidden in the coefficients somewhere. I’m also not a magician, so really who am I to say :)

kenjackson 4y ago

Does it really provide the statistically most common next token? I don’t think that’s even true. And we’ve seen what looks like emergent behavior as parameters increase. I suspect it is actually creating abstractions at a deeper level than standard DNNs.

graypegg 4y ago
I’d be curious to know! I get skeptical when it starts moving into the “emergent properties of language” but language was built by fleshy neurons, it could be that some inherent pattern emerges in the language we speak that reflects the inner workings of our brain, which by trying to emulate makes a similar brain on the other end. However, that feels a bit magical to me at the moment. Still interesting to be along for the ride humanity is going on!