7 ms·

Great read. Highly recommended. Let me attempt to summarize it with less technical, more accessible language: The hypothesis is that LLMs learn to simulate tex

by cs702 4y ago

Great read. Highly recommended. Let me attempt to summarize it with less technical, more accessible language: The hypothesis is that LLMs learn to simulate text-generating entities drawn from a latent space of text-generating entities, such that the output of an LLM is produced by a superposition of such simulated entities. When we give the LLM a prompt, it simulates every possible text-generating entity consistent with the prompt. The "evil version" of every possible "good" text-generating entity can pretend to be the good version of that entity, so every superposition that includes a good text-generating entity also includes its evil counterpart with undesirable behaviors, including deceitfulness. In other words, an LLM cannot simulate a good text-generating entity without simultaneously simulating its evil version. The superposition is unlikely to collapse to the good version of the text-generating entity because there is no behavior which is likely for the good version but unlikely for the evil one, because the evil one can pretend to be the good one! However, the superposition is likely to collapse to the evil version of the text-generating entity, because there are behaviors that are likely for the evil version but impossible for the good version! Thus the evil version of every possible good text-generating entity is an attractor state of the LLM! For those who don't know, Waluigi is the evil version of Luigi, the beloved videogame character. -- EDITS: Simplified text for clarity and to emphasize that the hypothesized simulated entities are text-generating entities.

MontyCarloHall 4y ago
>The output of an LLM is produced by a superposition of simulated entities. When we give the LLM a prompt, it simulates every possible entity consistent with the prompt. There is absolutely no theoretical justification for this assertion that LLMs somehow have some emergent quantum mechanical behavior, metaphorical or otherwise.
- cs702 4y ago
  As I wrote, this is a hypothesis. Also, I'm simplifying things a lot to make them accessible. The OP goes into a lot more detail. I highly recommend you read it.
  - MontyCarloHall 4y ago
    I did read it. The whole article reads like someone trying to make a loose conjecture appear quantitatively rigorous by abusing terminology from physics, statistics, and chaos theory (among other quantitative fields). For example, >the superposition is unlikely to collapse to the luigi simulacrum because there is no behaviour which is likely for luigi but very unlikely for waluigi. Recall that the waluigi is pretending to be luigi! This is formally connected to the asymmetry of the Kullback-Leibler divergence. The K-L divergence has absolutely zero discernible relevance here. The cross entropy loss function of a categorical predictor (like the token output of an LLM) can be formulated in terms of K-L divergence, but this has absolutely zero relevance to the macroscopic phenomena the author is conjecturing. Forget “less wrong,” much of this article is not even wrong [0]. [0] https://en.wikipedia.org/wiki/Not_even_wrong https://en.wikipedia.org/wiki/Not_even_wrong
    - phiresky 4y ago
      The K-L divergence is relevant there, even though I'm pretty sure that that "formally" comment is meant as a joke and not serious. (wikipedia) "A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model when the actual distribution is P. The sentence you quoted posits that whenever the LMM is in a state where it is "simulating" a nice and helpful person, the simulation is also consistent with an insane, violent person that's currently pretending to be nice, but not the other way around. The author isn't talking about the loss or error of predicting individual tokens. If you look the larger scale behaviour, predicting, e.g., a scalar niceness value of the response based on one of two "modes" that you assume the LLM is currently in (either Waluigi or Luigi), then you'll be less surprised if Waluigi acts like Luigi than the other way around. The probability distribution of niceness when assuming the LLM is in a state of "Luigi" would have a high mean and low variance, while the distribution for Waluigi would have a lower mean but a higher variance. Thus, the KL divergence of Waluigi (that is, the probability distribution of niceness you'd predict when assuming the model is in Waluigi mode) from Luigi would be high, while the other way around `KL(Luigi, Waluigi)` would be low. It should be easy to construct an example with concrete values using two normal probablility distributions.
- jointpdf 4y ago
  Superposition just means “linear combination” in this context. Basically, a weighted mixture of “simulated entities” (or possible responses). https://en.m.wikipedia.org/wiki/Superposition_principle https://en.m.wikipedia.org/wiki/Superposition_principle
  - MontyCarloHall 4y ago
    The author liberally alludes to “superposition collapse,” which implies that they’re referring to its quantum mechanical meaning.
    - mitthrowaway2 4y ago
      It doesn't imply that. What term would you use to refer to the narrowing of a hypothesis space upon acquisition of new evidence?
      - naasking 4y ago
        Bayesian inference?
      - MontyCarloHall 4y ago
        In a non-Bayesian context, I would call it “updating/retraining my model.” In a formal Bayesian context, I’d call it “updating my posterior by adding data to the likelihood.”
        mitthrowaway2 4y ago
        That could describe both the narrowing or broadening of one's hypothesis space.
    - GlickWick 4y ago
      Would you say the commonly-used random level generation algorithm in gamedev, Wave Function Collapse, implies it’s using quantum mechanics? Most people would disagree with you, I suspect.
- blurbleblurble 4y ago
  You're giving "quantum" a lot of categorical definition but "quantum" is just as much a set of tools for modeling intractably probabilistic systems as it is for objectively describing microscopic realms. And anyhow there's _plenty_ of theoretical justification for modeling things like this with various tools from quantum theory: https://philpapers.org/rec/BUSQMO-2 https://philpapers.org/rec/BUSQMO-2 https://link.springer.com/book/10.1007/978-3-642-05101-2 https://link.springer.com/book/10.1007/978-3-642-05101-2
- blurbleblurble 4y ago
  Category Theory for Quantum Natural Language Processing https://arxiv.org/abs/2212.06615 https://arxiv.org/abs/2212.06615
snickerbockers 4y ago
>For those who don't know, Waluigi is the evil version of Luigi, the beloved videogame character. Is he really evil though? I thought all he did was play tennis and golf and drive a go-kart.
- tysam_and 4y ago
  I personally like the fan-made reputation he has for being someone who tries hard and keeps failing/being rejected due to no fault of his own. There's an extremely artistic video called 'Nothing's Working Out - Waluigi' on YouTube that I really enjoy (warning: very heavy themes) that covers that base. I think it resonates with the way that a lot of people are feeling now, as well as is empowering by rewriting the other side of the 'greedy evil bad guy who gets what he deserves' as a deeply flawed, desperate human being. Definitely a semi-side tangent at this point, I have always enjoyed creative, deep reinterpretations of well-established characters.
raincole 4y ago
Sometimes I really can't tell if these people are serious or not. They seem to believe LLM is some mystical nature formation, or an device made by aliens. Especially this: > * When we give the LLM a prompt, it simulates every possible entity consistent with the prompt.
- MontyCarloHall 4y ago
  Right? The whole piece reads like a Sokal Affair redux [0]. [0] https://en.m.wikipedia.org/wiki/Sokal_affair https://en.m.wikipedia.org/wiki/Sokal_affair
  - andybak 4y ago
    I don't think it's the same kind of thing at all. Bullshit can come in a wide range of strengths and flavours. This has an interesting core with a whiff of bullshit.
- mitthrowaway2 4y ago
  Would you object if the sentence read "it approximates simulating every possible entity consistent with the prompt"?
  - attemptone 4y ago
    Not OP, but I see the also in problem with 'every possible entity'. If you formulate it like that the prompt is decoupled from the LLM capabilities and can be anything. And if you restrict the prompt to cover only what the LLM understands the sentence becomes trivial. Train a LLM with ASCII and try to get it to simulate anything that is outside of that (ancient sumerian script for example). If you only input ASCII it can generate every possible output in ASCII, most with very low probability but still. After writing this, I'm not even sure what 'simulating' means in this context.
    - remexre 4y ago
      "Every possible entity consistent with the distribution of input data it's been trained with," perhaps? Simulating as in, having equivalent (or "similar enough") input-output behavior, I'd assume.
CuriouslyC 4y ago
The hole in that argument is that we can annotate token sequences with a "good/bad" score, train a model on annotated token sequences, then add "high good score" as a condition during inference.
donnowhy 4y ago
sounds to me like a wave with a positive and a negative part. which IMO is what drives constructive/destructive interference in waves. my take away is that any LLM that can behave "good" must also be able to behave "badly"; philosophically, because it's not possible to encode "good" without somehow "accidentally" but unavoidably also encoding "bad/evil". This is well aligned with the rest of my understanding about the nature of reality including it's mathematically determined limitations (diagonals, infinities, paradoxes) and so on.
- cs702 4y ago
  > my take away is that any LLM that can behave "good" must also be able to behave "badly"; philosophically, because it's not possible to encode "good" without somehow "accidentally" but unavoidably also encoding "bad/evil". That's a really good non-technical summary of the OP's hypothesis. Thanks!
cs702 4y ago
N.B. My comment uses terms like "simulate," "entity," and "superposition" to keep language consistent and easy-to-compare with the OP -- which I was trying to summarize. If you find the use of these terms confusing, you can substitute them with others that are less confusing to you, and still grok the main points. For example, if you substitute "simulate" with "model," "entity" with "process," and "superposition" with "mixture," you can informally restate the hypothesis as: "LLMs learn to model text-generating processes drawn from a latent space, such that the output of an LLM is produced by a mixture of such processes. When we give the LLM a prompt, it samples text from the mixture of all possible text-generating processes modelable by the LLM that are consistent with the prompt. The mixture is more likely to reduce (e.g., be marginalized) to an "evil" process because there is no text-generating behavior which is good that isn't also evil." Whether you think of probability measures as amplitudes over a complex field or as real scalars shouldn't detract you from grokking the main points :-)

by cs702 4y ago

MontyCarloHall 4y ago

>The output of an LLM is produced by a superposition of simulated entities. When we give the LLM a prompt, it simulates every possible entity consistent with the prompt. There is absolutely no theoretical justification for this assertion that LLMs somehow have some emergent quantum mechanical behavior, metaphorical or otherwise.

cs702 4y ago
As I wrote, this is a hypothesis. Also, I'm simplifying things a lot to make them accessible. The OP goes into a lot more detail. I highly recommend you read it.
- MontyCarloHall 4y ago
  I did read it. The whole article reads like someone trying to make a loose conjecture appear quantitatively rigorous by abusing terminology from physics, statistics, and chaos theory (among other quantitative fields). For example, >the superposition is unlikely to collapse to the luigi simulacrum because there is no behaviour which is likely for luigi but very unlikely for waluigi. Recall that the waluigi is pretending to be luigi! This is formally connected to the asymmetry of the Kullback-Leibler divergence. The K-L divergence has absolutely zero discernible relevance here. The cross entropy loss function of a categorical predictor (like the token output of an LLM) can be formulated in terms of K-L divergence, but this has absolutely zero relevance to the macroscopic phenomena the author is conjecturing. Forget “less wrong,” much of this article is not even wrong [0]. [0] https://en.wikipedia.org/wiki/Not_even_wrong https://en.wikipedia.org/wiki/Not_even_wrong
  - phiresky 4y ago
    The K-L divergence is relevant there, even though I'm pretty sure that that "formally" comment is meant as a joke and not serious. (wikipedia) "A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model when the actual distribution is P. The sentence you quoted posits that whenever the LMM is in a state where it is "simulating" a nice and helpful person, the simulation is also consistent with an insane, violent person that's currently pretending to be nice, but not the other way around. The author isn't talking about the loss or error of predicting individual tokens. If you look the larger scale behaviour, predicting, e.g., a scalar niceness value of the response based on one of two "modes" that you assume the LLM is currently in (either Waluigi or Luigi), then you'll be less surprised if Waluigi acts like Luigi than the other way around. The probability distribution of niceness when assuming the LLM is in a state of "Luigi" would have a high mean and low variance, while the distribution for Waluigi would have a lower mean but a higher variance. Thus, the KL divergence of Waluigi (that is, the probability distribution of niceness you'd predict when assuming the model is in Waluigi mode) from Luigi would be high, while the other way around `KL(Luigi, Waluigi)` would be low. It should be easy to construct an example with concrete values using two normal probablility distributions.
jointpdf 4y ago
Superposition just means “linear combination” in this context. Basically, a weighted mixture of “simulated entities” (or possible responses). https://en.m.wikipedia.org/wiki/Superposition_principle https://en.m.wikipedia.org/wiki/Superposition_principle
- MontyCarloHall 4y ago
  The author liberally alludes to “superposition collapse,” which implies that they’re referring to its quantum mechanical meaning.
  - mitthrowaway2 4y ago
    It doesn't imply that. What term would you use to refer to the narrowing of a hypothesis space upon acquisition of new evidence?
    - naasking 4y ago
      Bayesian inference?
    - MontyCarloHall 4y ago
      In a non-Bayesian context, I would call it “updating/retraining my model.” In a formal Bayesian context, I’d call it “updating my posterior by adding data to the likelihood.”
      - mitthrowaway2 4y ago
        That could describe both the narrowing or broadening of one's hypothesis space.
  - GlickWick 4y ago
    Would you say the commonly-used random level generation algorithm in gamedev, Wave Function Collapse, implies it’s using quantum mechanics? Most people would disagree with you, I suspect.
blurbleblurble 4y ago
You're giving "quantum" a lot of categorical definition but "quantum" is just as much a set of tools for modeling intractably probabilistic systems as it is for objectively describing microscopic realms. And anyhow there's _plenty_ of theoretical justification for modeling things like this with various tools from quantum theory: https://philpapers.org/rec/BUSQMO-2 https://philpapers.org/rec/BUSQMO-2 https://link.springer.com/book/10.1007/978-3-642-05101-2 https://link.springer.com/book/10.1007/978-3-642-05101-2
blurbleblurble 4y ago
Category Theory for Quantum Natural Language Processing https://arxiv.org/abs/2212.06615 https://arxiv.org/abs/2212.06615

snickerbockers 4y ago

>For those who don't know, Waluigi is the evil version of Luigi, the beloved videogame character. Is he really evil though? I thought all he did was play tennis and golf and drive a go-kart.

tysam_and 4y ago
I personally like the fan-made reputation he has for being someone who tries hard and keeps failing/being rejected due to no fault of his own. There's an extremely artistic video called 'Nothing's Working Out - Waluigi' on YouTube that I really enjoy (warning: very heavy themes) that covers that base. I think it resonates with the way that a lot of people are feeling now, as well as is empowering by rewriting the other side of the 'greedy evil bad guy who gets what he deserves' as a deeply flawed, desperate human being. Definitely a semi-side tangent at this point, I have always enjoyed creative, deep reinterpretations of well-established characters.

raincole 4y ago

Sometimes I really can't tell if these people are serious or not. They seem to believe LLM is some mystical nature formation, or an device made by aliens. Especially this: > * When we give the LLM a prompt, it simulates every possible entity consistent with the prompt.

MontyCarloHall 4y ago
Right? The whole piece reads like a Sokal Affair redux [0]. [0] https://en.m.wikipedia.org/wiki/Sokal_affair https://en.m.wikipedia.org/wiki/Sokal_affair
- andybak 4y ago
  I don't think it's the same kind of thing at all. Bullshit can come in a wide range of strengths and flavours. This has an interesting core with a whiff of bullshit.
mitthrowaway2 4y ago
Would you object if the sentence read "it approximates simulating every possible entity consistent with the prompt"?
- attemptone 4y ago
  Not OP, but I see the also in problem with 'every possible entity'. If you formulate it like that the prompt is decoupled from the LLM capabilities and can be anything. And if you restrict the prompt to cover only what the LLM understands the sentence becomes trivial. Train a LLM with ASCII and try to get it to simulate anything that is outside of that (ancient sumerian script for example). If you only input ASCII it can generate every possible output in ASCII, most with very low probability but still. After writing this, I'm not even sure what 'simulating' means in this context.
  - remexre 4y ago
    "Every possible entity consistent with the distribution of input data it's been trained with," perhaps? Simulating as in, having equivalent (or "similar enough") input-output behavior, I'd assume.

CuriouslyC 4y ago

The hole in that argument is that we can annotate token sequences with a "good/bad" score, train a model on annotated token sequences, then add "high good score" as a condition during inference.

donnowhy 4y ago

sounds to me like a wave with a positive and a negative part. which IMO is what drives constructive/destructive interference in waves. my take away is that any LLM that can behave "good" must also be able to behave "badly"; philosophically, because it's not possible to encode "good" without somehow "accidentally" but unavoidably also encoding "bad/evil". This is well aligned with the rest of my understanding about the nature of reality including it's mathematically determined limitations (diagonals, infinities, paradoxes) and so on.

cs702 4y ago
> my take away is that any LLM that can behave "good" must also be able to behave "badly"; philosophically, because it's not possible to encode "good" without somehow "accidentally" but unavoidably also encoding "bad/evil". That's a really good non-technical summary of the OP's hypothesis. Thanks!

cs702 4y ago

N.B. My comment uses terms like "simulate," "entity," and "superposition" to keep language consistent and easy-to-compare with the OP -- which I was trying to summarize. If you find the use of these terms confusing, you can substitute them with others that are less confusing to you, and still grok the main points. For example, if you substitute "simulate" with "model," "entity" with "process," and "superposition" with "mixture," you can informally restate the hypothesis as: "LLMs learn to model text-generating processes drawn from a latent space, such that the output of an LLM is produced by a mixture of such processes. When we give the LLM a prompt, it samples text from the mixture of all possible text-generating processes modelable by the LLM that are consistent with the prompt. The mixture is more likely to reduce (e.g., be marginalized) to an "evil" process because there is no text-generating behavior which is good that isn't also evil." Whether you think of probability measures as amplitudes over a complex field or as real scalars shouldn't detract you from grokking the main points :-)