5 ms·

> GPT4 can't know the real reason why it wrote anything. It can never tell you the real reason why it made a mistake. Define "know" such that humans have it an

> GPT4 can't know the real reason why it wrote anything. It can never tell you the real reason why it made a mistake. Define "know" such that humans have it and GPT-4 does not have it.

foolrush 3y ago
Epistemology 101. Statistical pattern matching has no epistemological truth beyond a correspondence of linguistic statistics. It is merely statistical probability, which hardly can be classified as an epistemological basis. If we pretend for a moment it is, the ontologies formed are most certainly peculiar, and we can expect ideologies that emerge out of the ontologies are problematic.
- williamcotton 3y ago
  Beyesians accept an epistemological foundation of statistical priors updated by experience. Predictive processing is well established in the neuroscience community. Capital T Truth is really of very little interest to anyone outside of faith-based epistemology like religion.
  - foolrush 3y ago
    The issue is that the definition of “statistics” is anchored in a magnitude of frequency of glyphs. The “information” is fabricated in this regard, pulled up out of the ether, and by decree christened as “meaning”. Numbers carry no meaning, nor do the magnitudes arbitrarily assigned to meaning. The map is not the territory.
    - stevenhuang 3y ago
      Have you heard of information theory? Numbers can mean anything. A multitude of numbers as voltage potentials and ion gradients sufficiently describe your brain. Biology manifests this arrangement as a brain, without which this arrangement would also be similarly meaningless. Your argument against deriving meaning from statistics completely ignores that the brain also works this way.
      - dS0rrow 3y ago
        > Have you heard of information theory? > Your argument against deriving meaning from statistics completely ignores that the brain also works this way. The brain is not predicting it's compressing.
        stevenhuang 3y ago
        The current prevailing theory in neuroscience is in fact the brain is a prediction engine. https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Predictive_coding
        foolrush 3y ago
        [flagged]
        stevenhuang 3y ago
        My former claim is that everything in the natural world can be reduced to statistics, so saying meaning cannot be derived "because statistics" is a very poor argument. The second is a theory for the underlying mechanisms of the brain. I'm sorry you don't understand.
        naasking 3y ago
        > The brain is not predicting it's compressing Compression requires prediction, therefore your brain requires prediction.
        dS0rrow 3y ago
        Some form of prediction being used by the higher-level neurons doesn't make the brain a prediction engine.
        naasking 3y ago
        I'm not sure who claimed the brain was a "predictive engine" or what that means exactly. The OP specifically referenced predictive coding which describes precisely what is meant, and has empirical support. If you meant this as a comparison to machine learning, then a predictive coding model closely matches.
    - naasking 3y ago
      > The “information” is fabricated in this regard, pulled up out of the ether, and by decree christened as “meaning”. No, not fabricated, but inferred from a structured corpus of information generated by other semantic processes (humans). > Numbers carry no meaning, nor do the magnitudes arbitrarily assigned to meaning. Prove it. > The map is not the territory. Except if the territory is information, in which case the map is literally the territory. Knowledge is information, is it not?
  - discreteevent 3y ago
    Physicists are very interested in capital T Truth. They work with explanations not just statistical priors.
    - dS0rrow 3y ago
      This talk by Feynman is slightly related: https://youtu.be/obCjODeoLVw https://youtu.be/obCjODeoLVw
- stevenhuang 3y ago
  Statical pattern matching is how human brains work.
- naasking 3y ago
  > Epistemology 101. Statistical pattern matching has no epistemological truth beyond a correspondence of linguistic statistics. It is merely statistical Your belief in epistemic truth is a statistical inference from your perception of apparently reliable causality. How do you ground this inductive inference? Your attempt to appeal to epistemology 101 with a casual dismissal as "mere statistical probability" covers this deep, gaping maw. Bayesian inference reduces to classical logic when all probabilities are pinned to 0 and 1, but in what circumstances can we actually demonstrably infer absolute certainty? None that I can think of, except one's own existence.
  - foolrush 3y ago
    I did not lay claim to epistemic truth. I referenced epistemology. That is, there are a plurality of epistemologies, all of which have their own epistemic truth mechanics. So I would agree with you, entirely! I was merely pointing out that statistical correlation alone affords no epistemological basis.
skybrian 3y ago
Okay, this isn't quite what you asked but it's close: we can decide to go on an errand and remember what we were doing even if we didn't write it down. (Though sometimes we do forget, so shopping lists are useful.) This doesn't mean our justifications are accurate, but at least we do sometimes have "debug logs," some memories (perhaps faint and inaccurate) of internal thoughts that we can use to try to reconstruct what we were thinking. An LLM cannot remember anything it didn't write down. It can't do that at all. They would need to change the API.
- TeMPOraL 3y ago
  LLM doesn't have any other runtime state but for what it wrote down (assuming the user even provides it). For comparing with humans, I think LLMs are not like our minds, but like the inner voice. Chat history is the working memory.
  - skybrian 3y ago
    Yep, and we have the text. It’s very transparent that way. But it might be worth thinking about what a satisfying explanation of the LLM’s behavior would look like. It would have to be a high-level summary of the calculations that were done to predict each new token. Knowing which previous tokens got attention and what higher-level concepts it calculated would help. (Also, if the temperature isn’t zero, there’s a random number generator involved.) The LLM doesn’t have access to that, once the calculation is done. It has the text, same as us. It can guess what might have happened, same as us. When it guesses it’s going to anthropomorphize by picking an explanation that would make sense for a human, because it’s trained to write justifications that a human would write.
- naasking 3y ago
  > An LLM cannot remember anything it didn't write down. It can't do that at all. Ok, but I'm not sure what that has to do with knowledge. You can ask ChatGPT to reflect on and correct its own answers, so it clearly has access to its own outputs, which is a form of memory.
  - 3y ago

> GPT4 can't know the real reason why it wrote anything. It can never tell you the real reason why it made a mistake. Define "know" such that humans have it and GPT-4 does not have it.

foolrush 3y ago

Epistemology 101. Statistical pattern matching has no epistemological truth beyond a correspondence of linguistic statistics. It is merely statistical probability, which hardly can be classified as an epistemological basis. If we pretend for a moment it is, the ontologies formed are most certainly peculiar, and we can expect ideologies that emerge out of the ontologies are problematic.

williamcotton 3y ago
Beyesians accept an epistemological foundation of statistical priors updated by experience. Predictive processing is well established in the neuroscience community. Capital T Truth is really of very little interest to anyone outside of faith-based epistemology like religion.
- foolrush 3y ago
  The issue is that the definition of “statistics” is anchored in a magnitude of frequency of glyphs. The “information” is fabricated in this regard, pulled up out of the ether, and by decree christened as “meaning”. Numbers carry no meaning, nor do the magnitudes arbitrarily assigned to meaning. The map is not the territory.
  - stevenhuang 3y ago
    Have you heard of information theory? Numbers can mean anything. A multitude of numbers as voltage potentials and ion gradients sufficiently describe your brain. Biology manifests this arrangement as a brain, without which this arrangement would also be similarly meaningless. Your argument against deriving meaning from statistics completely ignores that the brain also works this way.
    - dS0rrow 3y ago
      > Have you heard of information theory? > Your argument against deriving meaning from statistics completely ignores that the brain also works this way. The brain is not predicting it's compressing.
      - stevenhuang 3y ago
        The current prevailing theory in neuroscience is in fact the brain is a prediction engine. https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Predictive_coding
        foolrush 3y ago
        [flagged]
        stevenhuang 3y ago
        My former claim is that everything in the natural world can be reduced to statistics, so saying meaning cannot be derived "because statistics" is a very poor argument. The second is a theory for the underlying mechanisms of the brain. I'm sorry you don't understand.
      - naasking 3y ago
        > The brain is not predicting it's compressing Compression requires prediction, therefore your brain requires prediction.
        dS0rrow 3y ago
        Some form of prediction being used by the higher-level neurons doesn't make the brain a prediction engine.
        naasking 3y ago
        I'm not sure who claimed the brain was a "predictive engine" or what that means exactly. The OP specifically referenced predictive coding which describes precisely what is meant, and has empirical support. If you meant this as a comparison to machine learning, then a predictive coding model closely matches.
  - naasking 3y ago
    > The “information” is fabricated in this regard, pulled up out of the ether, and by decree christened as “meaning”. No, not fabricated, but inferred from a structured corpus of information generated by other semantic processes (humans). > Numbers carry no meaning, nor do the magnitudes arbitrarily assigned to meaning. Prove it. > The map is not the territory. Except if the territory is information, in which case the map is literally the territory. Knowledge is information, is it not?
- discreteevent 3y ago
  Physicists are very interested in capital T Truth. They work with explanations not just statistical priors.
  - dS0rrow 3y ago
    This talk by Feynman is slightly related: https://youtu.be/obCjODeoLVw https://youtu.be/obCjODeoLVw
stevenhuang 3y ago
Statical pattern matching is how human brains work.
naasking 3y ago
> Epistemology 101. Statistical pattern matching has no epistemological truth beyond a correspondence of linguistic statistics. It is merely statistical Your belief in epistemic truth is a statistical inference from your perception of apparently reliable causality. How do you ground this inductive inference? Your attempt to appeal to epistemology 101 with a casual dismissal as "mere statistical probability" covers this deep, gaping maw. Bayesian inference reduces to classical logic when all probabilities are pinned to 0 and 1, but in what circumstances can we actually demonstrably infer absolute certainty? None that I can think of, except one's own existence.
- foolrush 3y ago
  I did not lay claim to epistemic truth. I referenced epistemology. That is, there are a plurality of epistemologies, all of which have their own epistemic truth mechanics. So I would agree with you, entirely! I was merely pointing out that statistical correlation alone affords no epistemological basis.

skybrian 3y ago

Okay, this isn't quite what you asked but it's close: we can decide to go on an errand and remember what we were doing even if we didn't write it down. (Though sometimes we do forget, so shopping lists are useful.) This doesn't mean our justifications are accurate, but at least we do sometimes have "debug logs," some memories (perhaps faint and inaccurate) of internal thoughts that we can use to try to reconstruct what we were thinking. An LLM cannot remember anything it didn't write down. It can't do that at all. They would need to change the API.

TeMPOraL 3y ago
LLM doesn't have any other runtime state but for what it wrote down (assuming the user even provides it). For comparing with humans, I think LLMs are not like our minds, but like the inner voice. Chat history is the working memory.
- skybrian 3y ago
  Yep, and we have the text. It’s very transparent that way. But it might be worth thinking about what a satisfying explanation of the LLM’s behavior would look like. It would have to be a high-level summary of the calculations that were done to predict each new token. Knowing which previous tokens got attention and what higher-level concepts it calculated would help. (Also, if the temperature isn’t zero, there’s a random number generator involved.) The LLM doesn’t have access to that, once the calculation is done. It has the text, same as us. It can guess what might have happened, same as us. When it guesses it’s going to anthropomorphize by picking an explanation that would make sense for a human, because it’s trained to write justifications that a human would write.
naasking 3y ago
> An LLM cannot remember anything it didn't write down. It can't do that at all. Ok, but I'm not sure what that has to do with knowledge. You can ask ChatGPT to reflect on and correct its own answers, so it clearly has access to its own outputs, which is a form of memory.
- 3y ago