5 ms·

>The output of an LLM is produced by a superposition of simulated entities. When we give the LLM a prompt, it simulates every possible entity consistent with th

>The output of an LLM is produced by a superposition of simulated entities. When we give the LLM a prompt, it simulates every possible entity consistent with the prompt. There is absolutely no theoretical justification for this assertion that LLMs somehow have some emergent quantum mechanical behavior, metaphorical or otherwise.

cs702 4y ago
As I wrote, this is a hypothesis. Also, I'm simplifying things a lot to make them accessible. The OP goes into a lot more detail. I highly recommend you read it.
- MontyCarloHall 4y ago
  I did read it. The whole article reads like someone trying to make a loose conjecture appear quantitatively rigorous by abusing terminology from physics, statistics, and chaos theory (among other quantitative fields). For example, >the superposition is unlikely to collapse to the luigi simulacrum because there is no behaviour which is likely for luigi but very unlikely for waluigi. Recall that the waluigi is pretending to be luigi! This is formally connected to the asymmetry of the Kullback-Leibler divergence. The K-L divergence has absolutely zero discernible relevance here. The cross entropy loss function of a categorical predictor (like the token output of an LLM) can be formulated in terms of K-L divergence, but this has absolutely zero relevance to the macroscopic phenomena the author is conjecturing. Forget “less wrong,” much of this article is not even wrong [0]. [0] https://en.wikipedia.org/wiki/Not_even_wrong https://en.wikipedia.org/wiki/Not_even_wrong
  - phiresky 4y ago
    The K-L divergence is relevant there, even though I'm pretty sure that that "formally" comment is meant as a joke and not serious. (wikipedia) "A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model when the actual distribution is P. The sentence you quoted posits that whenever the LMM is in a state where it is "simulating" a nice and helpful person, the simulation is also consistent with an insane, violent person that's currently pretending to be nice, but not the other way around. The author isn't talking about the loss or error of predicting individual tokens. If you look the larger scale behaviour, predicting, e.g., a scalar niceness value of the response based on one of two "modes" that you assume the LLM is currently in (either Waluigi or Luigi), then you'll be less surprised if Waluigi acts like Luigi than the other way around. The probability distribution of niceness when assuming the LLM is in a state of "Luigi" would have a high mean and low variance, while the distribution for Waluigi would have a lower mean but a higher variance. Thus, the KL divergence of Waluigi (that is, the probability distribution of niceness you'd predict when assuming the model is in Waluigi mode) from Luigi would be high, while the other way around `KL(Luigi, Waluigi)` would be low. It should be easy to construct an example with concrete values using two normal probablility distributions.
jointpdf 4y ago
Superposition just means “linear combination” in this context. Basically, a weighted mixture of “simulated entities” (or possible responses). https://en.m.wikipedia.org/wiki/Superposition_principle https://en.m.wikipedia.org/wiki/Superposition_principle
- MontyCarloHall 4y ago
  The author liberally alludes to “superposition collapse,” which implies that they’re referring to its quantum mechanical meaning.
  - mitthrowaway2 4y ago
    It doesn't imply that. What term would you use to refer to the narrowing of a hypothesis space upon acquisition of new evidence?
    - naasking 4y ago
      Bayesian inference?
    - MontyCarloHall 4y ago
      In a non-Bayesian context, I would call it “updating/retraining my model.” In a formal Bayesian context, I’d call it “updating my posterior by adding data to the likelihood.”
      - mitthrowaway2 4y ago
        That could describe both the narrowing or broadening of one's hypothesis space.
  - GlickWick 4y ago
    Would you say the commonly-used random level generation algorithm in gamedev, Wave Function Collapse, implies it’s using quantum mechanics? Most people would disagree with you, I suspect.
blurbleblurble 4y ago
You're giving "quantum" a lot of categorical definition but "quantum" is just as much a set of tools for modeling intractably probabilistic systems as it is for objectively describing microscopic realms. And anyhow there's _plenty_ of theoretical justification for modeling things like this with various tools from quantum theory: https://philpapers.org/rec/BUSQMO-2 https://philpapers.org/rec/BUSQMO-2 https://link.springer.com/book/10.1007/978-3-642-05101-2 https://link.springer.com/book/10.1007/978-3-642-05101-2
blurbleblurble 4y ago
Category Theory for Quantum Natural Language Processing https://arxiv.org/abs/2212.06615 https://arxiv.org/abs/2212.06615

cs702 4y ago

As I wrote, this is a hypothesis. Also, I'm simplifying things a lot to make them accessible. The OP goes into a lot more detail. I highly recommend you read it.

MontyCarloHall 4y ago
I did read it. The whole article reads like someone trying to make a loose conjecture appear quantitatively rigorous by abusing terminology from physics, statistics, and chaos theory (among other quantitative fields). For example, >the superposition is unlikely to collapse to the luigi simulacrum because there is no behaviour which is likely for luigi but very unlikely for waluigi. Recall that the waluigi is pretending to be luigi! This is formally connected to the asymmetry of the Kullback-Leibler divergence. The K-L divergence has absolutely zero discernible relevance here. The cross entropy loss function of a categorical predictor (like the token output of an LLM) can be formulated in terms of K-L divergence, but this has absolutely zero relevance to the macroscopic phenomena the author is conjecturing. Forget “less wrong,” much of this article is not even wrong [0]. [0] https://en.wikipedia.org/wiki/Not_even_wrong https://en.wikipedia.org/wiki/Not_even_wrong
- phiresky 4y ago
  The K-L divergence is relevant there, even though I'm pretty sure that that "formally" comment is meant as a joke and not serious. (wikipedia) "A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model when the actual distribution is P. The sentence you quoted posits that whenever the LMM is in a state where it is "simulating" a nice and helpful person, the simulation is also consistent with an insane, violent person that's currently pretending to be nice, but not the other way around. The author isn't talking about the loss or error of predicting individual tokens. If you look the larger scale behaviour, predicting, e.g., a scalar niceness value of the response based on one of two "modes" that you assume the LLM is currently in (either Waluigi or Luigi), then you'll be less surprised if Waluigi acts like Luigi than the other way around. The probability distribution of niceness when assuming the LLM is in a state of "Luigi" would have a high mean and low variance, while the distribution for Waluigi would have a lower mean but a higher variance. Thus, the KL divergence of Waluigi (that is, the probability distribution of niceness you'd predict when assuming the model is in Waluigi mode) from Luigi would be high, while the other way around `KL(Luigi, Waluigi)` would be low. It should be easy to construct an example with concrete values using two normal probablility distributions.

jointpdf 4y ago

Superposition just means “linear combination” in this context. Basically, a weighted mixture of “simulated entities” (or possible responses). https://en.m.wikipedia.org/wiki/Superposition_principle https://en.m.wikipedia.org/wiki/Superposition_principle

MontyCarloHall 4y ago
The author liberally alludes to “superposition collapse,” which implies that they’re referring to its quantum mechanical meaning.
- mitthrowaway2 4y ago
  It doesn't imply that. What term would you use to refer to the narrowing of a hypothesis space upon acquisition of new evidence?
  - naasking 4y ago
    Bayesian inference?
  - MontyCarloHall 4y ago
    In a non-Bayesian context, I would call it “updating/retraining my model.” In a formal Bayesian context, I’d call it “updating my posterior by adding data to the likelihood.”
    - mitthrowaway2 4y ago
      That could describe both the narrowing or broadening of one's hypothesis space.
- GlickWick 4y ago
  Would you say the commonly-used random level generation algorithm in gamedev, Wave Function Collapse, implies it’s using quantum mechanics? Most people would disagree with you, I suspect.

blurbleblurble 4y ago

You're giving "quantum" a lot of categorical definition but "quantum" is just as much a set of tools for modeling intractably probabilistic systems as it is for objectively describing microscopic realms. And anyhow there's _plenty_ of theoretical justification for modeling things like this with various tools from quantum theory: https://philpapers.org/rec/BUSQMO-2 https://philpapers.org/rec/BUSQMO-2 https://link.springer.com/book/10.1007/978-3-642-05101-2 https://link.springer.com/book/10.1007/978-3-642-05101-2

blurbleblurble 4y ago

Category Theory for Quantum Natural Language Processing https://arxiv.org/abs/2212.06615 https://arxiv.org/abs/2212.06615