4 ms·

Interesting to hear him say this: > And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firin

Interesting to hear him say this: > And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn't make sense to me, but that just shows that I'm naive. I think people maybe miss that LLM output does involve a ‘loop’ back - maybe even a ‘strange’ loop back, and I’m surprised to see Hofstadter himself fail to pick up on it. When you run an LLM on a context and sample from its output, you take that sampled output it generated, update the context, and iterate. So the LLM is not just feeding one way - it’s taking its output, adding it to its input, and then going round again. So I don’t think this implies what Hofstadter is saying about intelligence maybe being less complex than he thought.

jebarker 3y ago
I was surprised he said this too. Even without the autoregressive part of GPT models you have a deep transformer with attention, so even a single forward pass can modify its own intermediate outputs.
- PartiallyTyped 3y ago
  The interesting thing though is that attention effectively allows a model to meta-learn based on the current context, so in many ways, it may be thought of as analogous to a brain without long term memory.
- yldedly 3y ago
  What do you mean? Attention is just matrix multiplication and softmax, it's all feed forward.
  - jebarker 3y ago
    I wasn't disputing that it's feed-forward. I just meant that stacked transformer layers can be thought of as an iterative refinement of the intermediate activations. Not the same as an autoregressive process that receives previous outputs as inputs, but far more expressive than a single transformer layer.
gamegoblin 3y ago
Just being a part of any auto-regressive system does not contradict his statement. Go look at the GPT training code, here is the exact line: https://github.com/karpathy/nanoGPT/blob/master/train.py#L122 https://github.com/karpathy/nanoGPT/blob/master/train.py#L12... The model is only trained to predict the next token. The training regime is purely next-token prediction. There is no loopiness whatsoever here, strange or ordinary. Just because you take that feedforward neural network and wrap it in a loop to feed it its own output does not change the architecture of the neural net itself. The neural network was trained in one direction and runs in one direction. Hofstadter is surprised that such an architecture yields something that looks like intelligence. He specifically used the correct term "feedforward" to constrast with recurrent neural networks, which GPT is not: https://en.wikipedia.org/wiki/Feedforward_neural_network https://en.wikipedia.org/wiki/Feedforward_neural_network
- jameshart 3y ago
  It only yields something that looks like intelligence when you update the context and iterate though.
  - gamegoblin 3y ago
    GPT can give a single Yes/No answer that indicates a fair amount of intelligence for the right question. No iteration there. Just a single pass through the network. Hofstadter is surprised by this.
    - jameshart 3y ago
      Well, no, it can produce a probability distribution over all possible tokens, among which ‘yes’ or ‘oui’ or ‘はい’ or ‘totally’ or ‘no’ or ‘nein’ or the beginning of ‘as a large language model I am unable to answer that question’ are all represented. Which is either more or less impressive than just being able to answer ‘yes or no’ depending on your priors I guess. There’s maybe an interesting philosophical question of perspective there because if you think of the GPT as answering the question ‘if you had just read this, what token would you expect to read next?’ That doesn’t seem like a question that necessarily requires ‘intelligence’ so much as ‘data’. It’s just a classification problem and we’ve been throwing NNs at that for years. But if you ask the question ‘if you had just written this, what token would you expect to output next?’ It feels like the answer would require intelligence. But maybe they’re basically identical questions?

jebarker 3y ago

I was surprised he said this too. Even without the autoregressive part of GPT models you have a deep transformer with attention, so even a single forward pass can modify its own intermediate outputs.

PartiallyTyped 3y ago
The interesting thing though is that attention effectively allows a model to meta-learn based on the current context, so in many ways, it may be thought of as analogous to a brain without long term memory.
yldedly 3y ago
What do you mean? Attention is just matrix multiplication and softmax, it's all feed forward.
- jebarker 3y ago
  I wasn't disputing that it's feed-forward. I just meant that stacked transformer layers can be thought of as an iterative refinement of the intermediate activations. Not the same as an autoregressive process that receives previous outputs as inputs, but far more expressive than a single transformer layer.

gamegoblin 3y ago

Just being a part of any auto-regressive system does not contradict his statement. Go look at the GPT training code, here is the exact line: https://github.com/karpathy/nanoGPT/blob/master/train.py#L122 https://github.com/karpathy/nanoGPT/blob/master/train.py#L12... The model is only trained to predict the next token. The training regime is purely next-token prediction. There is no loopiness whatsoever here, strange or ordinary. Just because you take that feedforward neural network and wrap it in a loop to feed it its own output does not change the architecture of the neural net itself. The neural network was trained in one direction and runs in one direction. Hofstadter is surprised that such an architecture yields something that looks like intelligence. He specifically used the correct term "feedforward" to constrast with recurrent neural networks, which GPT is not: https://en.wikipedia.org/wiki/Feedforward_neural_network https://en.wikipedia.org/wiki/Feedforward_neural_network

jameshart 3y ago
It only yields something that looks like intelligence when you update the context and iterate though.
- gamegoblin 3y ago
  GPT can give a single Yes/No answer that indicates a fair amount of intelligence for the right question. No iteration there. Just a single pass through the network. Hofstadter is surprised by this.
  - jameshart 3y ago
    Well, no, it can produce a probability distribution over all possible tokens, among which ‘yes’ or ‘oui’ or ‘はい’ or ‘totally’ or ‘no’ or ‘nein’ or the beginning of ‘as a large language model I am unable to answer that question’ are all represented. Which is either more or less impressive than just being able to answer ‘yes or no’ depending on your priors I guess. There’s maybe an interesting philosophical question of perspective there because if you think of the GPT as answering the question ‘if you had just read this, what token would you expect to read next?’ That doesn’t seem like a question that necessarily requires ‘intelligence’ so much as ‘data’. It’s just a classification problem and we’ve been throwing NNs at that for years. But if you ask the question ‘if you had just written this, what token would you expect to output next?’ It feels like the answer would require intelligence. But maybe they’re basically identical questions?