4 ms·

We know that human introspection is at best imperfect, and at worst outright fiction, thanks to Sperry's split brain experiments, so while it might be interesti

by vidarh 2mo ago

We know that human introspection is at best imperfect, and at worst outright fiction, thanks to Sperry's split brain experiments, so while it might be interesting, if the chain of reasoning isn't indicative, that would be in line with how humans work.

ilovecake1984 2mo ago
This is a good take. People overwhelmingly fit a narrative to how they make decisions rather than make decisions according to a chain of reasoning.
- andy99 2mo ago
  That’s just conflating reasoning and “parallel reconstruction” - there is such thing as reasoning, and I agree it’s probably less common in decision making, but it’s fundamental to many tasks where we figure out a solution, like writing an algorithm, math proofs, etc.
  - pegasus 2mo ago
    Sure, but the same must be true for these models as well. They must be doing some form of reasoning, even if it's not formally correct and it relies instead on all kind of half-assed heuristics which only work inside certain unexamined boundaries, which leads to wrong conclusions when those boundaries are being trespassed, etc. And even if it's not the reasoning they report when asked. But all this can't be that different to how humans reason most of the time either. I suspect humans have other ways that help with error correction and guiding the reasoning effort, but that's another story.
    - fc417fc802 2mo ago
      > They must be doing some form of reasoning, even if it's not formally correct and ... Surely it is worthwhile to attempt to understand the details of that? And if we seek human equivalent performance then it is reasonable to wonder if the reasoning achieved to date is the "correct" sort.
  - vidarh 2mo ago
    There is explicit reasoning where we externalise steps, but it is slow and cumbersome. The point is that when we make a decision without externalising each step, and then afterward explain why and how we made that decision, there is no evidence that explanation is true, and a lot that it is a retrospective fiction.
  - ilovecake1984 2mo ago
    Of course, but it is rare.
  - joshjob42 2mo ago
    Maybe, but I suspect that a lot of the actual things that produce the algorithm, proof, etc are not actually explicit reasoning. Q: "Why did you try that vs some other thing at this point?" A:"seemed right to do"/"kinda looks like other stuff I've seen before"/"first thing that came to mind" etc. Why pursue this one avenue vs some other, typically its cause it seemed more promising, and the person can come up with reasons but did they explicitly verbalize a fully sound chain of thinking at the time? Probably not. Not to say that the explicit thinking, or writing things down, isn't important. But if I examine the process by which I develop a proof or something, there's a lot of vague hunches, blind alleys, etc that come along the way. And, many of the blind alleys etc probably aren't actually that important in the end for me finding the right answer -- if you were to trim that part out of my own internal reasoning trace but left the rest intact, I'd still get the right answer because, well, it was a blind alley.
    - vidarh 2mo ago
      Exactly. Very little of what we do is "explicit thinking", because it's slow to have to develop and verbalise those arguments step by step. Most of it is "hunches" that we then retroactively justify if asked, and those retroactive justifications were demonstrated not to be trustworthy by Sperry's experiments. We don't even have evidence to suggest that humans can reliably introspect our past decisions, but we do have evidence to suggest we readily manufacture them.
  - jameshart 2mo ago
    Is it really? I think the assumption that there is some sort of formal ‘reasoning’ process that is inherently superior to whatever ‘non-reason-based’ decision process ‘non-reasoning’ systems use is not immediately justifiable. Most reasoning is ultimately based in heuristics: this is the right thing to do next because it has worked in the past. This is the right thing to do because our intuition tells us doing it will lead to this desirable outcome. When you’re formulating a math proof, sure, the steps you are laying out make up a ‘reasoned’ argument but how you choose what steps to take is wholly intuitive. Why assume the contrapositive to begin this proof? It’s worked in the past and it might work here. It’s all pattern matching trained on a reward function.
    - Plont 2mo ago
      It's true that humans are not perfectly logical actors, but even so, most humans won't tell you seriously that the swimming pools in the Titanic are empty, or that glue is a great pizza topping. At the end of the day, there's a difference between showing your work for a math problem by writing down each logical step as you work through it, vs. just blindly copying both the answer and the "show your work" part from your classmates. That humans can fail at math, or at other kinds of logic problems, is immaterial. It's still an entirely different process, and it leads to different results. The fact that humans often fail because they're actively prioritizing something else can also be for the best. Like, it's reasonable to get a math problem wrong because you're distracted by argueing classmates who might be about to start a fight right next to you. Or you might realize that the premise of a question itself is wrong. Or you might logically know that the Earth goes around the sun, but decide to pretend otherwise because it's not worth dying over. Humans are always balancing priorities, and it doesn't always make sense to judge their success by a single metric like "do these math problems and show your work within 10 minutes". LLMs do not have these conflicts, even when we would want them to.
ScoobleDoodle 2mo ago
Even there half the brain was doing the real reasoning. It’s just the other half which was the verbal/story half didn’t know what that real reasoning was so made up a story it believed to make it fit but was different reasoning. It’s not without reasoning. But half got the reasoning wrong; even if thinking it was right.
- vidarh 2mo ago
  No, in the experiment in question, the experimenters made the choice, and claimed to the brain half that couldn't observe that the one that could made the choice. That is how we know the rationale wasn't the result of information making it between brain halves somehow. But irrespective of that, the point is exactly that it made up a rationale that couldn't possibly be true, and insisted it was. In other words, we have no reason to trust our own mind when we rationalize our own actions.

by vidarh 2mo ago

ilovecake1984 2mo ago

This is a good take. People overwhelmingly fit a narrative to how they make decisions rather than make decisions according to a chain of reasoning.

andy99 2mo ago
That’s just conflating reasoning and “parallel reconstruction” - there is such thing as reasoning, and I agree it’s probably less common in decision making, but it’s fundamental to many tasks where we figure out a solution, like writing an algorithm, math proofs, etc.
- pegasus 2mo ago
  Sure, but the same must be true for these models as well. They must be doing some form of reasoning, even if it's not formally correct and it relies instead on all kind of half-assed heuristics which only work inside certain unexamined boundaries, which leads to wrong conclusions when those boundaries are being trespassed, etc. And even if it's not the reasoning they report when asked. But all this can't be that different to how humans reason most of the time either. I suspect humans have other ways that help with error correction and guiding the reasoning effort, but that's another story.
  - fc417fc802 2mo ago
    > They must be doing some form of reasoning, even if it's not formally correct and ... Surely it is worthwhile to attempt to understand the details of that? And if we seek human equivalent performance then it is reasonable to wonder if the reasoning achieved to date is the "correct" sort.
- vidarh 2mo ago
  There is explicit reasoning where we externalise steps, but it is slow and cumbersome. The point is that when we make a decision without externalising each step, and then afterward explain why and how we made that decision, there is no evidence that explanation is true, and a lot that it is a retrospective fiction.
- ilovecake1984 2mo ago
  Of course, but it is rare.
- joshjob42 2mo ago
  Maybe, but I suspect that a lot of the actual things that produce the algorithm, proof, etc are not actually explicit reasoning. Q: "Why did you try that vs some other thing at this point?" A:"seemed right to do"/"kinda looks like other stuff I've seen before"/"first thing that came to mind" etc. Why pursue this one avenue vs some other, typically its cause it seemed more promising, and the person can come up with reasons but did they explicitly verbalize a fully sound chain of thinking at the time? Probably not. Not to say that the explicit thinking, or writing things down, isn't important. But if I examine the process by which I develop a proof or something, there's a lot of vague hunches, blind alleys, etc that come along the way. And, many of the blind alleys etc probably aren't actually that important in the end for me finding the right answer -- if you were to trim that part out of my own internal reasoning trace but left the rest intact, I'd still get the right answer because, well, it was a blind alley.
  - vidarh 2mo ago
    Exactly. Very little of what we do is "explicit thinking", because it's slow to have to develop and verbalise those arguments step by step. Most of it is "hunches" that we then retroactively justify if asked, and those retroactive justifications were demonstrated not to be trustworthy by Sperry's experiments. We don't even have evidence to suggest that humans can reliably introspect our past decisions, but we do have evidence to suggest we readily manufacture them.
- jameshart 2mo ago
  Is it really? I think the assumption that there is some sort of formal ‘reasoning’ process that is inherently superior to whatever ‘non-reason-based’ decision process ‘non-reasoning’ systems use is not immediately justifiable. Most reasoning is ultimately based in heuristics: this is the right thing to do next because it has worked in the past. This is the right thing to do because our intuition tells us doing it will lead to this desirable outcome. When you’re formulating a math proof, sure, the steps you are laying out make up a ‘reasoned’ argument but how you choose what steps to take is wholly intuitive. Why assume the contrapositive to begin this proof? It’s worked in the past and it might work here. It’s all pattern matching trained on a reward function.
  - Plont 2mo ago
    It's true that humans are not perfectly logical actors, but even so, most humans won't tell you seriously that the swimming pools in the Titanic are empty, or that glue is a great pizza topping. At the end of the day, there's a difference between showing your work for a math problem by writing down each logical step as you work through it, vs. just blindly copying both the answer and the "show your work" part from your classmates. That humans can fail at math, or at other kinds of logic problems, is immaterial. It's still an entirely different process, and it leads to different results. The fact that humans often fail because they're actively prioritizing something else can also be for the best. Like, it's reasonable to get a math problem wrong because you're distracted by argueing classmates who might be about to start a fight right next to you. Or you might realize that the premise of a question itself is wrong. Or you might logically know that the Earth goes around the sun, but decide to pretend otherwise because it's not worth dying over. Humans are always balancing priorities, and it doesn't always make sense to judge their success by a single metric like "do these math problems and show your work within 10 minutes". LLMs do not have these conflicts, even when we would want them to.

ScoobleDoodle 2mo ago

Even there half the brain was doing the real reasoning. It’s just the other half which was the verbal/story half didn’t know what that real reasoning was so made up a story it believed to make it fit but was different reasoning. It’s not without reasoning. But half got the reasoning wrong; even if thinking it was right.

vidarh 2mo ago
No, in the experiment in question, the experimenters made the choice, and claimed to the brain half that couldn't observe that the one that could made the choice. That is how we know the rationale wasn't the result of information making it between brain halves somehow. But irrespective of that, the point is exactly that it made up a rationale that couldn't possibly be true, and insisted it was. In other words, we have no reason to trust our own mind when we rationalize our own actions.