5 ms·

> it is very rare for GPT4 not to be able to understand why it went wrong. That's not what's going on. GPT4 can't know the real reason why it wrote anything. I

> it is very rare for GPT4 not to be able to understand why it went wrong. That's not what's going on. GPT4 can't know the real reason why it wrote anything. It can never tell you the real reason why it made a mistake. Instead, we can say that GPT4 is fairly good at inventing plausible explanations about why someone did something, but it's no more accurate explaining its own mistakes than it would be at explaining one of your mistakes. [1] A better explanation of what happened is that asking for an explanation resulted in chain-of-thought reasoning, and by telling it that its previous answer was wrong, it was biased to pick a different answer. But this is often fragile. You'd want to do the same test multiple times, and compare regular chain-of-thought reasoning with what happens when you explicitly tell it that a particular answer is wrong. [1] https://skybrian.substack.com/p/ai-chatbots-dont-know-why-they-did https://skybrian.substack.com/p/ai-chatbots-dont-know-why-th...

famouswaffles 4y ago
>That's not what's going on. GPT4 can't know the real reason why it wrote anything. Boy do I have news for you. People can't recreate previous mental states, it's all plausible post-hoc rationalization. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ And the rationalizations aren't always grounded either. The brain is just fine making up completely bogus explanations you believe to be true but couldn't possibly be so. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522/ https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf The conscious self is at best a co-pilot. A Co-pilot that has shockingly little insight into the operations of the plane but thinks he knows much more. Even your senses gets some pretty big post-processing by your brain you're consciously unaware of. For example, ever wondered why the first second seems longer when you suddenly look at a clock ? Human beings may not have the best visual acuity, but we are quite good at distinguishing colors and exceptional at processing visual information. A significant portion of our brain is dedicated to processing the visual stimuli from our binocular vision. Nevertheless, binocular vision necessitates that both eyes focus on a single point. Our eyes can track a focused target as it moves, but they are unable to smoothly scan across our visual field. This is too difficult to process, so our eyes naturally jump from one point to another. Yet, during these jumps, known as "saccades," our brain does not even attempt to comprehend the blurry sensations that come in. In effect, you temporarily go completely blind! However, you don't feel like you went blind, right? Your vision doesn't go black or anything, and it seems like you could see the entire time. This is because our brains fill in the lag while our eyes are moving, creating the illusion that we were seeing something instead of being blind during that time. Strangely enough, our brains fill it in with *what we see when they stop*! In other words, our brains go back and edit our memory of a split second ago to make it seem like we were looking at our new point of focus the whole time our eyes were moving to actually see it. This phenomenon occurs when you look at the second hand of a clock. When you glance over at the clock, your brain tells you that you were looking at it for longer than you actually were, adding the time your eyes were moving but not yet seeing the clock. If the second hand was moving during this time, your brain will believe the second hand was in its new position longer than it actually was. Based on this flawed memory, it will appear as if the second hand stayed in that position for longer than a second.
- skybrian 4y ago
  Yes, of course. We often don't know our real motivations either, our justifications are often self-serving and made up, and more generally there's a lot we don't know about how the brain works. But we do at least have some memories of our own thought processes and some experience observing our own reactions to things. You can get better at introspection. Attempting to understand and explain your own mistakes is useful. An LLM won't be able to do this sort of learning without changing its training or architecture somehow.
  - famouswaffles 4y ago
    Actually just added a long paragraph showing how even memories or real time sense data is pleasant fabrication. Also every time you remember something, you're not really recalling it. Your brain rewrites your memory every time you "remember" often introducing false data or inconsistencies. It's one of the reasons human memory is so fallible and inadmissible. That said, I don't disagree with the overall point of what you've said. Attempted Introspection is useful. The point I'm making is to not dismiss LLM "introspection" simply because it's rationalization. It can be extremely useful too.
    - skybrian 4y ago
      Fair enough! The part I see missing for an LLM is learning from feedback about whether its justifications "work" for explaining its own actions. Keep in mind that an LLM is not human and its justifications often shouldn't look like human justifications. But it's going to be scored by how human its justifications look, so you will never get that. (With the exception of things explicitly trained in like knowing its own cut-off date - that's quite a non-human justification for not knowing something.) Also, it's easy to come up with examples showing that we do remember what we were thinking, even if we didn't say it out loud. A simple example might be running an errand. Of course, sometimes we do forget, which is why a shopping list is useful. An LLM cannot do that at all. It writes it down or it's lost. It can only pretend to remember an internal thought it had.
      - 4y ago
naasking 4y ago
> GPT4 can't know the real reason why it wrote anything. It can never tell you the real reason why it made a mistake. Define "know" such that humans have it and GPT-4 does not have it.
- foolrush 4y ago
  Epistemology 101. Statistical pattern matching has no epistemological truth beyond a correspondence of linguistic statistics. It is merely statistical probability, which hardly can be classified as an epistemological basis. If we pretend for a moment it is, the ontologies formed are most certainly peculiar, and we can expect ideologies that emerge out of the ontologies are problematic.
  - williamcotton 4y ago
    Beyesians accept an epistemological foundation of statistical priors updated by experience. Predictive processing is well established in the neuroscience community. Capital T Truth is really of very little interest to anyone outside of faith-based epistemology like religion.
    - foolrush 4y ago
      The issue is that the definition of “statistics” is anchored in a magnitude of frequency of glyphs. The “information” is fabricated in this regard, pulled up out of the ether, and by decree christened as “meaning”. Numbers carry no meaning, nor do the magnitudes arbitrarily assigned to meaning. The map is not the territory.
      - stevenhuang 4y ago
        Have you heard of information theory? Numbers can mean anything. A multitude of numbers as voltage potentials and ion gradients sufficiently describe your brain. Biology manifests this arrangement as a brain, without which this arrangement would also be similarly meaningless. Your argument against deriving meaning from statistics completely ignores that the brain also works this way.
two_in_one 4y ago
> It can never tell you the real reason why it made a mistake. From my experience it can. Sometimes it writes the code and I have to point out that it produces a wrong result probably because of some function's behavior. GPT-4 can figure out the rest and fix it. However, in general, it can go circles giving conflicting statements.
- jiggywiggy 4y ago
  OP knows it looks like that but is trying to explain what might cause that instead of genuine understanding.
- skybrian 4y ago
  That's slightly different. It can sometimes spot a bug in some code it wrote, in the same way it could spot a bug in the code someone else wrote. It could also give a human-like justification for why the bug is there. But that's different from a real explanation for why it wrote the code with a bug. A real explanation would have to be rather non-human.
TexanFeller 4y ago
Hugo Mercier was recently on Sean Carroll's podcast talking about the concepts from his new book The Enigma of Reason. The gist of it is that most of the function of what we call reasoning doesn't exist to implement logical reasoning, but rather evolved to provide the social utility of providing "reasons" to convince or placate other humans. I guess what I'm saying is we may be closer to ChatGPT than we think - we may not usually be logically reasoning when we provide reasons, but inventing after the fact reasons that are statistically likely to be accepted for what we did or what we want to do. Podcast: https://www.preposterousuniverse.com/podcast/2023/04/17/233-hugo-mercier-on-reasoning-and-skepticism/ https://www.preposterousuniverse.com/podcast/2023/04/17/233-...

famouswaffles 4y ago

>That's not what's going on. GPT4 can't know the real reason why it wrote anything. Boy do I have news for you. People can't recreate previous mental states, it's all plausible post-hoc rationalization. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ And the rationalizations aren't always grounded either. The brain is just fine making up completely bogus explanations you believe to be true but couldn't possibly be so. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522/ https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf The conscious self is at best a co-pilot. A Co-pilot that has shockingly little insight into the operations of the plane but thinks he knows much more. Even your senses gets some pretty big post-processing by your brain you're consciously unaware of. For example, ever wondered why the first second seems longer when you suddenly look at a clock ? Human beings may not have the best visual acuity, but we are quite good at distinguishing colors and exceptional at processing visual information. A significant portion of our brain is dedicated to processing the visual stimuli from our binocular vision. Nevertheless, binocular vision necessitates that both eyes focus on a single point. Our eyes can track a focused target as it moves, but they are unable to smoothly scan across our visual field. This is too difficult to process, so our eyes naturally jump from one point to another. Yet, during these jumps, known as "saccades," our brain does not even attempt to comprehend the blurry sensations that come in. In effect, you temporarily go completely blind! However, you don't feel like you went blind, right? Your vision doesn't go black or anything, and it seems like you could see the entire time. This is because our brains fill in the lag while our eyes are moving, creating the illusion that we were seeing something instead of being blind during that time. Strangely enough, our brains fill it in with *what we see when they stop*! In other words, our brains go back and edit our memory of a split second ago to make it seem like we were looking at our new point of focus the whole time our eyes were moving to actually see it. This phenomenon occurs when you look at the second hand of a clock. When you glance over at the clock, your brain tells you that you were looking at it for longer than you actually were, adding the time your eyes were moving but not yet seeing the clock. If the second hand was moving during this time, your brain will believe the second hand was in its new position longer than it actually was. Based on this flawed memory, it will appear as if the second hand stayed in that position for longer than a second.

skybrian 4y ago
Yes, of course. We often don't know our real motivations either, our justifications are often self-serving and made up, and more generally there's a lot we don't know about how the brain works. But we do at least have some memories of our own thought processes and some experience observing our own reactions to things. You can get better at introspection. Attempting to understand and explain your own mistakes is useful. An LLM won't be able to do this sort of learning without changing its training or architecture somehow.
- famouswaffles 4y ago
  Actually just added a long paragraph showing how even memories or real time sense data is pleasant fabrication. Also every time you remember something, you're not really recalling it. Your brain rewrites your memory every time you "remember" often introducing false data or inconsistencies. It's one of the reasons human memory is so fallible and inadmissible. That said, I don't disagree with the overall point of what you've said. Attempted Introspection is useful. The point I'm making is to not dismiss LLM "introspection" simply because it's rationalization. It can be extremely useful too.
  - skybrian 4y ago
    Fair enough! The part I see missing for an LLM is learning from feedback about whether its justifications "work" for explaining its own actions. Keep in mind that an LLM is not human and its justifications often shouldn't look like human justifications. But it's going to be scored by how human its justifications look, so you will never get that. (With the exception of things explicitly trained in like knowing its own cut-off date - that's quite a non-human justification for not knowing something.) Also, it's easy to come up with examples showing that we do remember what we were thinking, even if we didn't say it out loud. A simple example might be running an errand. Of course, sometimes we do forget, which is why a shopping list is useful. An LLM cannot do that at all. It writes it down or it's lost. It can only pretend to remember an internal thought it had.
    - 4y ago

naasking 4y ago

> GPT4 can't know the real reason why it wrote anything. It can never tell you the real reason why it made a mistake. Define "know" such that humans have it and GPT-4 does not have it.

foolrush 4y ago
Epistemology 101. Statistical pattern matching has no epistemological truth beyond a correspondence of linguistic statistics. It is merely statistical probability, which hardly can be classified as an epistemological basis. If we pretend for a moment it is, the ontologies formed are most certainly peculiar, and we can expect ideologies that emerge out of the ontologies are problematic.
- williamcotton 4y ago
  Beyesians accept an epistemological foundation of statistical priors updated by experience. Predictive processing is well established in the neuroscience community. Capital T Truth is really of very little interest to anyone outside of faith-based epistemology like religion.
  - foolrush 4y ago
    The issue is that the definition of “statistics” is anchored in a magnitude of frequency of glyphs. The “information” is fabricated in this regard, pulled up out of the ether, and by decree christened as “meaning”. Numbers carry no meaning, nor do the magnitudes arbitrarily assigned to meaning. The map is not the territory.
    - stevenhuang 4y ago
      Have you heard of information theory? Numbers can mean anything. A multitude of numbers as voltage potentials and ion gradients sufficiently describe your brain. Biology manifests this arrangement as a brain, without which this arrangement would also be similarly meaningless. Your argument against deriving meaning from statistics completely ignores that the brain also works this way.

two_in_one 4y ago

> It can never tell you the real reason why it made a mistake. From my experience it can. Sometimes it writes the code and I have to point out that it produces a wrong result probably because of some function's behavior. GPT-4 can figure out the rest and fix it. However, in general, it can go circles giving conflicting statements.

jiggywiggy 4y ago
OP knows it looks like that but is trying to explain what might cause that instead of genuine understanding.
skybrian 4y ago
That's slightly different. It can sometimes spot a bug in some code it wrote, in the same way it could spot a bug in the code someone else wrote. It could also give a human-like justification for why the bug is there. But that's different from a real explanation for why it wrote the code with a bug. A real explanation would have to be rather non-human.

TexanFeller 4y ago

Hugo Mercier was recently on Sean Carroll's podcast talking about the concepts from his new book The Enigma of Reason. The gist of it is that most of the function of what we call reasoning doesn't exist to implement logical reasoning, but rather evolved to provide the social utility of providing "reasons" to convince or placate other humans. I guess what I'm saying is we may be closer to ChatGPT than we think - we may not usually be logically reasoning when we provide reasons, but inventing after the fact reasons that are statistically likely to be accepted for what we did or what we want to do. Podcast: https://www.preposterousuniverse.com/podcast/2023/04/17/233-hugo-mercier-on-reasoning-and-skepticism/ https://www.preposterousuniverse.com/podcast/2023/04/17/233-...