3 ms·

>> Since LLMs infer statistical relationships of languages produced by human brains, they are in a real sense building a statistical model of how the human brai

>> Since LLMs infer statistical relationships of languages produced by human brains, they are in a real sense building a statistical model of how the human brain processes language. I think we understand very well that a statistical model can accurately predict the behaviour of a system and still have nothing to do with how the system operates internally. In a sense, that's the big advantage of building statistical models: you don't need to consider the internals of the system you're modelling.

naasking 3y ago
> I think we understand very well that a statistical model can accurately predict the behaviour of a system and still have nothing to do with how the system operates internally. I agree that's true of some statistical models, I don't think it's true of Bayesian models. Solomonoff Induction has shown that Bayesian inference will reproduce any underlying computable function given a suitable prior. And this should be obvious given this simple sketch of the argument: classical logic is just Bayesian inference with all probabilities pinned to 0 and 1, and a model of a system in classical logic is a model of how it operates. Therefore you just need to run Bayes' rule long enough for the probabilities to converge. Many papers have shown that LLMs perform Bayesian inference.
- deleted 3y ago
  [deleted]
- tester89 3y ago
  Would you be willing to link some of those papers? For my curiosity, thanks.
  - naasking 3y ago
    Examples from a quick search: What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization, https://arxiv.org/abs/2305.19420 https://arxiv.org/abs/2305.19420 An Explanation of In-context Learning as Implicit Bayesian Inference, https://arxiv.org/abs/2111.02080 https://arxiv.org/abs/2111.02080 A Latent Space Theory for Emergent Abilities in Large Language Models, https://arxiv.org/abs/2304.09960 https://arxiv.org/abs/2304.09960
- YeGoblynQueenne 3y ago
  >> And this should be obvious given this simple sketch of the argument: classical logic is just Bayesian inference with all probabilities pinned to 0 and 1, and a model of a system in classical logic is a model of how it operates. Therefore you just need to run Bayes' rule long enough for the probabilities to converge. I've heard this before and it's an argument that perhaps makes sense from a Bayesian point of view, but makes no sense from a logic point of view. To clarify, here's some relevant background on classical logic and probabilities. I will take First Order Logic (FOL) as an example of a classical logic, but the same observations hold true for e.g. propositional logic. In FOL semantics, an interpretation assigns truth values in {true, false} to the atomic formulae ("atoms") of a predicate. In Bayesian inference, a distribution assigns probabilities to the outcomes of an event, such that the sum of probabilities of all outcomes of an event is 1. Now, for "classical logic [to be] just Bayesian inference with all probabilities pinned to 0 and 1", it must mean that, given an interpretation I, all atoms true under I are assigned the probability "1" and all atoms false under I are assigned the probability "0". This means that the sum of probabilities of all atoms is higher than 1 (it is n times 1, where n is the cardinality of the set of atoms true under I). This makes for a non-Bayesian distribution [*]. So your simple argument sketch doesn't work. Find a better one. ______________ [*] As an aside, all this means that we assume outcomes to be analogous to atoms, events to predicates and distributions to interpretations. There may be a better mapping between concepts and some have been proposed, e.g. that assign probabilities to interpretations rather than atoms, but under FOL semantics the only thing that takes a truth value is atoms, and formulae formed of atoms, and so whatever framework comes out of such transformations has nothing to do with classical logic anymore. Btw, you can also try to assign different probabilities to atoms than 0 and 1, but then you're outside classical logic again. The fact of the matter is that logic and probabilities are incompatible and any attempt to reconcile the two is doomed to fail. You can speak the language of certainty, or the language of uncertainty, but not both at once.
  - naasking 3y ago
    I'm going to consolidate replies to all three of your comments here and only going to address specific salient points that I think are the crux of disagreements: Re: Bayesian inference and logic To clarify, I was informally referring to the standard correspondence with propositional logic [1], but generalizations of this to include quantifiers also seem possible [2]. That said, I agree that my simple sketch is not a good justification for Solomonoff Induction, so a poor attempt at a summary on my part. Re: A "system" is not a function. I don't know a formal definition of a system, but if you're talking about whatever the human brain is doing that's likely to be more easily modelled as a Turing machine. If you agree that a system can be modelled as a function, then there's a formal equivalence. That's what "is" means to me. Certainly you can insist on a definition for "is" that's more like a small-step operational semantics to more faithfully model internals, but for the purposes of this discussion, I don't think the internal details matter if the observable inputs and outputs are the same. If a human brain "is" equivalent to some Turing machine, then I can also say that it "is" equivalent to some term in the lambda calculus. Re: If, by any chance, you think of the brain as a universal Turing machine (UTM) The brain can be described by a finite state automaton due to the Bekenstein Bound. So it's not a universal Turing machine, strictly speaking. Certainly it would be interesting to know how the internals of the brain work in minute detail, but that's ambitious. Re: They have consistently failed because the pieces of the puzzle don't fit. I'm curious specifically what doesn't fit. I understand there's tension between logic and probability, and learning and human-like reasoning are obviously still burgeoning fields, but the rest seem fairly well explored with plenty of correspondences. Re: I think in modern machine learning people have simply given up on trying to make any coherent image out of the puzzle pieces and just bashed them until they yielded and are now wedged together in incongruous patterns that sort of show some kind of image, there. Which is kinda what you'd expect from evolution by natural selection. [1] https://en.wikipedia.org/wiki/Bayes%27_theorem#Correspondence_to_other_mathematical_frameworks https://en.wikipedia.org/wiki/Bayes%27_theorem#Correspondenc... [2] https://www.lesswrong.com/posts/W8YscokXMiDnLKJ96/bayesian-inference-on-1st-order-logic https://www.lesswrong.com/posts/W8YscokXMiDnLKJ96/bayesian-i...

naasking 3y ago

> I think we understand very well that a statistical model can accurately predict the behaviour of a system and still have nothing to do with how the system operates internally. I agree that's true of some statistical models, I don't think it's true of Bayesian models. Solomonoff Induction has shown that Bayesian inference will reproduce any underlying computable function given a suitable prior. And this should be obvious given this simple sketch of the argument: classical logic is just Bayesian inference with all probabilities pinned to 0 and 1, and a model of a system in classical logic is a model of how it operates. Therefore you just need to run Bayes' rule long enough for the probabilities to converge. Many papers have shown that LLMs perform Bayesian inference.

deleted 3y ago
[deleted]
tester89 3y ago
Would you be willing to link some of those papers? For my curiosity, thanks.
- naasking 3y ago
  Examples from a quick search: What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization, https://arxiv.org/abs/2305.19420 https://arxiv.org/abs/2305.19420 An Explanation of In-context Learning as Implicit Bayesian Inference, https://arxiv.org/abs/2111.02080 https://arxiv.org/abs/2111.02080 A Latent Space Theory for Emergent Abilities in Large Language Models, https://arxiv.org/abs/2304.09960 https://arxiv.org/abs/2304.09960
YeGoblynQueenne 3y ago
>> And this should be obvious given this simple sketch of the argument: classical logic is just Bayesian inference with all probabilities pinned to 0 and 1, and a model of a system in classical logic is a model of how it operates. Therefore you just need to run Bayes' rule long enough for the probabilities to converge. I've heard this before and it's an argument that perhaps makes sense from a Bayesian point of view, but makes no sense from a logic point of view. To clarify, here's some relevant background on classical logic and probabilities. I will take First Order Logic (FOL) as an example of a classical logic, but the same observations hold true for e.g. propositional logic. In FOL semantics, an interpretation assigns truth values in {true, false} to the atomic formulae ("atoms") of a predicate. In Bayesian inference, a distribution assigns probabilities to the outcomes of an event, such that the sum of probabilities of all outcomes of an event is 1. Now, for "classical logic [to be] just Bayesian inference with all probabilities pinned to 0 and 1", it must mean that, given an interpretation I, all atoms true under I are assigned the probability "1" and all atoms false under I are assigned the probability "0". This means that the sum of probabilities of all atoms is higher than 1 (it is n times 1, where n is the cardinality of the set of atoms true under I). This makes for a non-Bayesian distribution [*]. So your simple argument sketch doesn't work. Find a better one. ______________ [*] As an aside, all this means that we assume outcomes to be analogous to atoms, events to predicates and distributions to interpretations. There may be a better mapping between concepts and some have been proposed, e.g. that assign probabilities to interpretations rather than atoms, but under FOL semantics the only thing that takes a truth value is atoms, and formulae formed of atoms, and so whatever framework comes out of such transformations has nothing to do with classical logic anymore. Btw, you can also try to assign different probabilities to atoms than 0 and 1, but then you're outside classical logic again. The fact of the matter is that logic and probabilities are incompatible and any attempt to reconcile the two is doomed to fail. You can speak the language of certainty, or the language of uncertainty, but not both at once.
- naasking 3y ago
  I'm going to consolidate replies to all three of your comments here and only going to address specific salient points that I think are the crux of disagreements: Re: Bayesian inference and logic To clarify, I was informally referring to the standard correspondence with propositional logic [1], but generalizations of this to include quantifiers also seem possible [2]. That said, I agree that my simple sketch is not a good justification for Solomonoff Induction, so a poor attempt at a summary on my part. Re: A "system" is not a function. I don't know a formal definition of a system, but if you're talking about whatever the human brain is doing that's likely to be more easily modelled as a Turing machine. If you agree that a system can be modelled as a function, then there's a formal equivalence. That's what "is" means to me. Certainly you can insist on a definition for "is" that's more like a small-step operational semantics to more faithfully model internals, but for the purposes of this discussion, I don't think the internal details matter if the observable inputs and outputs are the same. If a human brain "is" equivalent to some Turing machine, then I can also say that it "is" equivalent to some term in the lambda calculus. Re: If, by any chance, you think of the brain as a universal Turing machine (UTM) The brain can be described by a finite state automaton due to the Bekenstein Bound. So it's not a universal Turing machine, strictly speaking. Certainly it would be interesting to know how the internals of the brain work in minute detail, but that's ambitious. Re: They have consistently failed because the pieces of the puzzle don't fit. I'm curious specifically what doesn't fit. I understand there's tension between logic and probability, and learning and human-like reasoning are obviously still burgeoning fields, but the rest seem fairly well explored with plenty of correspondences. Re: I think in modern machine learning people have simply given up on trying to make any coherent image out of the puzzle pieces and just bashed them until they yielded and are now wedged together in incongruous patterns that sort of show some kind of image, there. Which is kinda what you'd expect from evolution by natural selection. [1] https://en.wikipedia.org/wiki/Bayes%27_theorem#Correspondence_to_other_mathematical_frameworks https://en.wikipedia.org/wiki/Bayes%27_theorem#Correspondenc... [2] https://www.lesswrong.com/posts/W8YscokXMiDnLKJ96/bayesian-inference-on-1st-order-logic https://www.lesswrong.com/posts/W8YscokXMiDnLKJ96/bayesian-i...