5 ms·

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in wh

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1], but how and what LLMs learn exactly remain a black box. When we dig deeper into the mechanisms that drive LLMs, we might get closer to understanding why they work so well in some senses, and why they could be catastrophic in other cases (see: the past month of search-based developments). I find trying to understand and reverse-engineer LLMs to be a personally exciting endeavour. As LLMs get better in the near future, I sure hope our understanding of them can keep up as well! [1] https://distill.pub/2020/circuits/zoom-in/ https://distill.pub/2020/circuits/zoom-in/

bilsbie 4y ago
Near work! I’m still confused how it knows to use “an” if it hasn’t chosen the word after it yet?
sharemywin 4y ago
you might find this paper interesting: https://arxiv.org/abs/2202.05262 https://arxiv.org/abs/2202.05262 Locating and Editing Factual Associations in GPT
- dpaleka 4y ago
  That paper (ROME) was the most famous paper in the field last year :) See also new interesting developments breaking the connection between "Locating" and "Editing": https://arxiv.org/abs/2301.04213 https://arxiv.org/abs/2301.04213 Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
jerpint 4y ago
Do you think it would ever be possible to “maximize” a neuron with certain sentences? What’s so different with the gradient ascent techniques with convolutions?
mungoman2 4y ago
Convolution is part of the network design though. Would a fully connected network learn to convolute? Or would it turn out that convolution is not necessary?
- nerdponx 4y ago
  The interesting part here isn't the convolution itself, it's how convolutional layers turn out to like "filters" or "detectors" for individual features. This is explained very well in the distill.pub article linked by GP. We know the architecture of LLMs because we created it, but we don't yet have the same level of understanding about them, or the same quality of analytical tools for reasoning about them.
- redox99 4y ago
  Yes it would, or at least a similar operation. The point of using a CNN instead of a FCN is that you force it to learn in a certain way that prevents overfitting. But given a sufficient dataset, and proper data augmentation you would expect a FCN to be able to identify objects regardless of translation. It's just that a CNN would train easier and better, with a smaller network (a FCN doing convolutions would be very wasteful). That's why traditionally you would pick your architecture to help it learn in a certain way (images=cnn, text=rnn/lstm/gru). But the nice thing about transformers is that they are more general.
- xmcqdpt2 4y ago
  They do and in fact it's relatively straightforward to show empirically on eg MNIST. The problem is that you need a much much larger network in the FCN case and thus need way more data and way more data augmentation to get a good result that isn't overfit to hell. In the case of CNN the reason it works is that an image of an object X is still an image of object X if the X is shifted left or right. The property is translationally invariant. CNN are basically the simplest way to encode translational invariance.
  - candiodari 4y ago
    > CNN are basically the simplest way to encode translational invariance That's the geometric deep learning theory, isn't it? Do you know if there's a list somewhere of exactly what invariance has which ways to simulate it? Like an overview?
ly3xqhl8g9 4y ago
Could a "type system" for neural weights be developed? Given a self-driving system, to be able to statically check that the neurons have the "Person" type, the "Don't Run Over Person" type, and so forth. What happens if you "transplant" the weights for ' an' to another network, some kind of transfer learning but componentized, does it still predict as accurately? If neural networks could be assembled from "types" it would be much easier to trust them.
- simonh 4y ago
  The way an LLM decides which word to use next is by evaluating the weightings of all the preceding words with every candidate word to calculate a probability for each of them. So if it selects ‘an’ as the next word, it’s because the weighting connecting ‘an’ to all the preceding words, and their orders in the text and relationships with each other predicted it should have a high probability of occurring. So you can’t extract the weightings for ‘an’ discretely because those weightings encode its connection with all the other words and combinations and sequences or clusters of words it might ever be used with, including their weightings with other preceding words, and their relationships, etc, etc.
  - ly3xqhl8g9 4y ago
    Right, but if there is such a thing as the very plastically named "Jeniffer Aniston neuron" [1], and further more, group equivariant deep learning [2], maybe there is a way in which you can isolate a certain concept/"type", such as Person, Car, and so forth; perhaps not even isolate, but rehydrate the context of where the concept takes place: as a brain does in various word plays, as in Who's on First [3], etc. Come to think of it, when someone teaches me a new concept, the principle of mass conservation, for instance, in some sense they are transferring their embedding into my brain, further on I will relate to mass conservation through what that person taught me. The transfer is a very lossy process, sure, but a transfer with reintegration nonetheless. Perhaps "mortal computation" [4] is a requirement. [1] https://en.wikipedia.org/wiki/Grandmother_cell https://en.wikipedia.org/wiki/Grandmother_cell [2] https://www.youtube.com/playlist?list=PL8FnQMH2k7jzPrxqdYufoiYVHim8PyZWd https://www.youtube.com/playlist?list=PL8FnQMH2k7jzPrxqdYufo... [3] https://www.youtube.com/watch?v=kTcRRaXV-fg https://www.youtube.com/watch?v=kTcRRaXV-fg [4] Geoffrey Hinton, The Forward-Forward Algorithm: Some Preliminary Investigations, chapter 8, https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf
deleted 4y ago
[deleted]
HarHarVeryFunny 4y ago
I wonder if you could comment on this (related to question of how far ahead these "LLM"s are planning). This is Wharton professor Ethan Mollick playing with the new Bing chat, which seems considerably more advanced than ChatGPT (based on GPT-4 perhaps?). Here he asks it to write something using Kurt Vonnegut's rules of writing. https://twitter.com/emollick/status/1626084142239649792 https://twitter.com/emollick/status/1626084142239649792 It seems hard to explain how Bing/GPT could have generated the Vonnegut-inspired cake story, having ingested the rules, without planning the whole thing before generating the first word. It seems there's an awful lot more going on internally in these models than a mere word by word autoregressive generation. It seems the prompt (in this case including Vonnegut's rules) is ingested and creates a complex internal state that is then responsible for the coherency and content of the output. The fact that it necessarily has to generate the output one word at a time seems to be a bit misleading in terms of understanding when the actual "output prediction" takes place.
- gbasin 4y ago
  There is "long range" dependence, it's just only on the prompt: the conversation with the user and the hidden header (e.g. "Answer as ChatGPT, an intelligent AI, state your reasons, be succinct, etc."). That ends up being enough.
  - HarHarVeryFunny 4y ago
    Sure, but the point being discussed is that despite the word by word output, the output does not appear to be "chosen" on a word by word basis. OP investigated the case where the word "an" anticipates the following word ("an apple" vs "a pear").
    - sharemywin 4y ago
      I see 2 options: 1. we don't know what they(coding layer between bing and GPT) look up and store as a prompt aka working memory. 2. it can do the equivalent of receiving it's own prompt silently. I seen with code it outputs the step for the code then writes the code. so there's some kind of plan and execute going on. maybe it can do that in model some how

bilsbie 4y ago

Near work! I’m still confused how it knows to use “an” if it hasn’t chosen the word after it yet?

sharemywin 4y ago

you might find this paper interesting: https://arxiv.org/abs/2202.05262 https://arxiv.org/abs/2202.05262 Locating and Editing Factual Associations in GPT

dpaleka 4y ago
That paper (ROME) was the most famous paper in the field last year :) See also new interesting developments breaking the connection between "Locating" and "Editing": https://arxiv.org/abs/2301.04213 https://arxiv.org/abs/2301.04213 Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models

jerpint 4y ago

Do you think it would ever be possible to “maximize” a neuron with certain sentences? What’s so different with the gradient ascent techniques with convolutions?

mungoman2 4y ago

Convolution is part of the network design though. Would a fully connected network learn to convolute? Or would it turn out that convolution is not necessary?

nerdponx 4y ago
The interesting part here isn't the convolution itself, it's how convolutional layers turn out to like "filters" or "detectors" for individual features. This is explained very well in the distill.pub article linked by GP. We know the architecture of LLMs because we created it, but we don't yet have the same level of understanding about them, or the same quality of analytical tools for reasoning about them.
redox99 4y ago
Yes it would, or at least a similar operation. The point of using a CNN instead of a FCN is that you force it to learn in a certain way that prevents overfitting. But given a sufficient dataset, and proper data augmentation you would expect a FCN to be able to identify objects regardless of translation. It's just that a CNN would train easier and better, with a smaller network (a FCN doing convolutions would be very wasteful). That's why traditionally you would pick your architecture to help it learn in a certain way (images=cnn, text=rnn/lstm/gru). But the nice thing about transformers is that they are more general.
xmcqdpt2 4y ago
They do and in fact it's relatively straightforward to show empirically on eg MNIST. The problem is that you need a much much larger network in the FCN case and thus need way more data and way more data augmentation to get a good result that isn't overfit to hell. In the case of CNN the reason it works is that an image of an object X is still an image of object X if the X is shifted left or right. The property is translationally invariant. CNN are basically the simplest way to encode translational invariance.
- candiodari 4y ago
  > CNN are basically the simplest way to encode translational invariance That's the geometric deep learning theory, isn't it? Do you know if there's a list somewhere of exactly what invariance has which ways to simulate it? Like an overview?

ly3xqhl8g9 4y ago

Could a "type system" for neural weights be developed? Given a self-driving system, to be able to statically check that the neurons have the "Person" type, the "Don't Run Over Person" type, and so forth. What happens if you "transplant" the weights for ' an' to another network, some kind of transfer learning but componentized, does it still predict as accurately? If neural networks could be assembled from "types" it would be much easier to trust them.

simonh 4y ago
The way an LLM decides which word to use next is by evaluating the weightings of all the preceding words with every candidate word to calculate a probability for each of them. So if it selects ‘an’ as the next word, it’s because the weighting connecting ‘an’ to all the preceding words, and their orders in the text and relationships with each other predicted it should have a high probability of occurring. So you can’t extract the weightings for ‘an’ discretely because those weightings encode its connection with all the other words and combinations and sequences or clusters of words it might ever be used with, including their weightings with other preceding words, and their relationships, etc, etc.
- ly3xqhl8g9 4y ago
  Right, but if there is such a thing as the very plastically named "Jeniffer Aniston neuron" [1], and further more, group equivariant deep learning [2], maybe there is a way in which you can isolate a certain concept/"type", such as Person, Car, and so forth; perhaps not even isolate, but rehydrate the context of where the concept takes place: as a brain does in various word plays, as in Who's on First [3], etc. Come to think of it, when someone teaches me a new concept, the principle of mass conservation, for instance, in some sense they are transferring their embedding into my brain, further on I will relate to mass conservation through what that person taught me. The transfer is a very lossy process, sure, but a transfer with reintegration nonetheless. Perhaps "mortal computation" [4] is a requirement. [1] https://en.wikipedia.org/wiki/Grandmother_cell https://en.wikipedia.org/wiki/Grandmother_cell [2] https://www.youtube.com/playlist?list=PL8FnQMH2k7jzPrxqdYufoiYVHim8PyZWd https://www.youtube.com/playlist?list=PL8FnQMH2k7jzPrxqdYufo... [3] https://www.youtube.com/watch?v=kTcRRaXV-fg https://www.youtube.com/watch?v=kTcRRaXV-fg [4] Geoffrey Hinton, The Forward-Forward Algorithm: Some Preliminary Investigations, chapter 8, https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf

deleted 4y ago

[deleted]

HarHarVeryFunny 4y ago

I wonder if you could comment on this (related to question of how far ahead these "LLM"s are planning). This is Wharton professor Ethan Mollick playing with the new Bing chat, which seems considerably more advanced than ChatGPT (based on GPT-4 perhaps?). Here he asks it to write something using Kurt Vonnegut's rules of writing. https://twitter.com/emollick/status/1626084142239649792 https://twitter.com/emollick/status/1626084142239649792 It seems hard to explain how Bing/GPT could have generated the Vonnegut-inspired cake story, having ingested the rules, without planning the whole thing before generating the first word. It seems there's an awful lot more going on internally in these models than a mere word by word autoregressive generation. It seems the prompt (in this case including Vonnegut's rules) is ingested and creates a complex internal state that is then responsible for the coherency and content of the output. The fact that it necessarily has to generate the output one word at a time seems to be a bit misleading in terms of understanding when the actual "output prediction" takes place.

gbasin 4y ago
There is "long range" dependence, it's just only on the prompt: the conversation with the user and the hidden header (e.g. "Answer as ChatGPT, an intelligent AI, state your reasons, be succinct, etc."). That ends up being enough.
- HarHarVeryFunny 4y ago
  Sure, but the point being discussed is that despite the word by word output, the output does not appear to be "chosen" on a word by word basis. OP investigated the case where the word "an" anticipates the following word ("an apple" vs "a pear").
  - sharemywin 4y ago
    I see 2 options: 1. we don't know what they(coding layer between bing and GPT) look up and store as a prompt aka working memory. 2. it can do the equivalent of receiving it's own prompt silently. I seen with code it outputs the step for the code then writes the code. so there's some kind of plan and execute going on. maybe it can do that in model some how