5 ms·

We know how they work, that is true. We don't know why they work, because if we could, then we could extrapolate what happens when you throw more compute at the

We know how they work, that is true. We don't know why they work, because if we could, then we could extrapolate what happens when you throw more compute at them, and no one would have been surprised about the capabilities of GPT-N+1. Also no one would have been caught with their pants down by seeing people jailbreak their models. To illustrate it in a different way: on a mechanistic level, we know how animal brains work, as well. Ganglions, calcium channels, the stuff. That doesn't help understand high level phenomena like cognition, which is the part that matters. If you're right about the LLMs revealing their inner working, that would be indeed a reason to chill out. But I have my doubts, given that LLMs are good at hallucinating. Could you justify why the human readability is actually true, and support that with examples?

lukev 4y ago
I don't need examples. It's simply how they work. This is why they hallucinate. A LLM is fundamentally a mathematical function (albeit a very complex one, with billions of terms (a.k.a parameters or weights)). The function does one thing and one thing only: it takes a sequence of tokens as input (the context), and it emits the next token(word)[1]. This is a stateless process: it has no "memory" and the model parameters are immutable; they are not changed during the generation process. In order to generate longer sequences of text, you call the function multiple times, each time appending the previously generated token to the input sequence. The output of the function is 100% dependent on the input. Therefore, the only "internal state" a model has is the input sequence, which is human-readable sequence of tokens. It can't "hallucinate", it can't "lie", and it can't "tell the truth", it can only emit tokens one at a time. It can't have a hidden "intent" without emitting those tokens, it can't "believe" something different than what it emits. [1] Actually a set of probabilities for the next token, and one is selected at random based on the "heat" generating setting, but this is irrelevant for the high-level view.
- rhn_mk1 4y ago
  I don't see how this proves that asking the model about its internal state will reveal its inner high level processes in a human-readable way. Perhaps there's a research paper which would explain it better?
- lukev 4y ago
  Responding here since it won't let me continue your thread any more. No, there's a fundamental misunderstanding here. I'm not saying the model will tell you the truth about its internal state if you ask it (it absolutely will not.) I'm saying it has no internal state, and no inner high level processes at all other than it's pre-baked, immutable parameters.
  - rhn_mk1 4y ago
    Then you did not read my post carefully enough. The question was not about "internal state" but "inner workings". The model clearly does something. The problem is that we don't know how to describe in human terms what happens between the matrix multiplication and the words it spits out. Whether it has state is completely irrelevant.
  - lukev 4y ago
    Whether it has inner state is highly relevant to my claim, which was that the only state a LLM has (aside from its parameters) is transparent and readable in English. Which the context is.
    - rhn_mk1 4y ago
      You're the one who put state in the conversation. State is part of the whole, and not the whole. It's not enough to understand state if you want to understand why they work. I feel like you're trying to muddy the waters by redirecting the problem to be about the state - it isn't.
      - lukev 4y ago
        On one hand I hate to belabor this point, but on the other I think it's actually super important. Both of things things are true: 1. The relationships between parameter weights are mysterious, non-evident, and we don't know precisely why it is so effective at token generation. 2. An agent built on top of a LLM cannot have any thought, intent, consideration, agenda, or idea that is not readable in plain english. Because all of those concepts involve state.
- nicpottier 4y ago
  This is a great reminder, thank you. As someone not skilled in this art, is there anything preventing us from opening that context window many orders of magnitude? What happens then? And what happens if it is then "thinking in text" faster than we can read them? (with an intent towards paper clips) This is a genuine question, I'm not trolling.
  - lukev 4y ago
    You could do those things in theory. I'm not saying that you could never build AGI on top of a LLM, or that such a AGI could not become "misaligned." I'm just saying that having a mental state that's natively in English is a nice property if one is worried about what they are "thinking."
  - ldhough 4y ago
    I'm very much not an expert either but apparently for the regular "attention" mechanism memory and compute requirements scale quadratically with respect to input sequence length. So increasing it by just two orders of magnitude would mean (I think) a full context window needs 10000x more memory and compute time and presumably costs would go up by at least as much. GPT3 (I think) uses the regular attention mechanism while 4 is unknown. However, GPT4 claims there are techniques to improve scaling (complexity down to sub-quadratic or linear) without affecting accuracy too much (I have no clue if true): sparse attention, long-range arena, reformer, and performer. I'm also pretty sure I've read (and anecdotally it seems true) that accuracy decreases with longer input/output sequences regardless. How much I also don't know.
Jensson 4y ago
> Also no one would have been caught with their pants down by seeing people jailbreak their models. Preventing jailbreak in a language model is like preventing a GO AI from drawing a dick with the pieces. You can try, but since the model doesn't have any concept of what you want it to do it is very hard to control that. Doesn't make the model smart, it just means that the model wasn't made to understand dick pictures.
- rhn_mk1 4y ago
  It does not make the model smart, but it demonstrates our inablity to control it despite wanting it. That strongly suggests that it's not fully understood.

lukev 4y ago

I don't need examples. It's simply how they work. This is why they hallucinate. A LLM is fundamentally a mathematical function (albeit a very complex one, with billions of terms (a.k.a parameters or weights)). The function does one thing and one thing only: it takes a sequence of tokens as input (the context), and it emits the next token(word)[1]. This is a stateless process: it has no "memory" and the model parameters are immutable; they are not changed during the generation process. In order to generate longer sequences of text, you call the function multiple times, each time appending the previously generated token to the input sequence. The output of the function is 100% dependent on the input. Therefore, the only "internal state" a model has is the input sequence, which is human-readable sequence of tokens. It can't "hallucinate", it can't "lie", and it can't "tell the truth", it can only emit tokens one at a time. It can't have a hidden "intent" without emitting those tokens, it can't "believe" something different than what it emits. [1] Actually a set of probabilities for the next token, and one is selected at random based on the "heat" generating setting, but this is irrelevant for the high-level view.

rhn_mk1 4y ago
I don't see how this proves that asking the model about its internal state will reveal its inner high level processes in a human-readable way. Perhaps there's a research paper which would explain it better?
lukev 4y ago
Responding here since it won't let me continue your thread any more. No, there's a fundamental misunderstanding here. I'm not saying the model will tell you the truth about its internal state if you ask it (it absolutely will not.) I'm saying it has no internal state, and no inner high level processes at all other than it's pre-baked, immutable parameters.
- rhn_mk1 4y ago
  Then you did not read my post carefully enough. The question was not about "internal state" but "inner workings". The model clearly does something. The problem is that we don't know how to describe in human terms what happens between the matrix multiplication and the words it spits out. Whether it has state is completely irrelevant.
- lukev 4y ago
  Whether it has inner state is highly relevant to my claim, which was that the only state a LLM has (aside from its parameters) is transparent and readable in English. Which the context is.
  - rhn_mk1 4y ago
    You're the one who put state in the conversation. State is part of the whole, and not the whole. It's not enough to understand state if you want to understand why they work. I feel like you're trying to muddy the waters by redirecting the problem to be about the state - it isn't.
    - lukev 4y ago
      On one hand I hate to belabor this point, but on the other I think it's actually super important. Both of things things are true: 1. The relationships between parameter weights are mysterious, non-evident, and we don't know precisely why it is so effective at token generation. 2. An agent built on top of a LLM cannot have any thought, intent, consideration, agenda, or idea that is not readable in plain english. Because all of those concepts involve state.
nicpottier 4y ago
This is a great reminder, thank you. As someone not skilled in this art, is there anything preventing us from opening that context window many orders of magnitude? What happens then? And what happens if it is then "thinking in text" faster than we can read them? (with an intent towards paper clips) This is a genuine question, I'm not trolling.
- lukev 4y ago
  You could do those things in theory. I'm not saying that you could never build AGI on top of a LLM, or that such a AGI could not become "misaligned." I'm just saying that having a mental state that's natively in English is a nice property if one is worried about what they are "thinking."
- ldhough 4y ago
  I'm very much not an expert either but apparently for the regular "attention" mechanism memory and compute requirements scale quadratically with respect to input sequence length. So increasing it by just two orders of magnitude would mean (I think) a full context window needs 10000x more memory and compute time and presumably costs would go up by at least as much. GPT3 (I think) uses the regular attention mechanism while 4 is unknown. However, GPT4 claims there are techniques to improve scaling (complexity down to sub-quadratic or linear) without affecting accuracy too much (I have no clue if true): sparse attention, long-range arena, reformer, and performer. I'm also pretty sure I've read (and anecdotally it seems true) that accuracy decreases with longer input/output sequences regardless. How much I also don't know.

Jensson 4y ago

> Also no one would have been caught with their pants down by seeing people jailbreak their models. Preventing jailbreak in a language model is like preventing a GO AI from drawing a dick with the pieces. You can try, but since the model doesn't have any concept of what you want it to do it is very hard to control that. Doesn't make the model smart, it just means that the model wasn't made to understand dick pictures.

rhn_mk1 4y ago
It does not make the model smart, but it demonstrates our inablity to control it despite wanting it. That strongly suggests that it's not fully understood.