4 ms·

These models produce correct answers to many problems that require “reasoning” (for any sensible meaning of the word), that are not in their training set.

by abm53 3y ago

all2 3y ago
That just shows that parroting will solve a number of problems that require reasoning, that is "reasoning" as we think of it can be reduced to a statistical process. Kids parrot their parents before they understand the meaning of what they are doing or saying. Meaning arises from a much more complex process than seeing/repeating. I think the same will be true with LLMs. True reasoning capabilities will be another revolution entirely. --- This just occurred to me; LLM's have a cargo cult level understanding of anything they've been trained on. Correct answers are actually statistical flukes -- purposeful because that's how we trained the models -- but not actually significant in terms of reasoning.
- a_wild_dandan 3y ago
  How do we test this alleged distinction between "true reasoning" and statistical parroting? What experiments can we perform on SotA models to make this idea falsifiable? Commonplace hypotheses like these give me strong "No True Scotsman" vibes, but I'm often wrong. Let's agree on a method, and I'll test it.
  - Art9681 3y ago
    There is none. No one understands what intelligence really means or self-awareness. Which is why they feel threatened by anything that challenges their self proclaimed uniquely human trait. The goal posts WILL move a lot more before this debate is settled. Inevitably the success rate and accuracy of LLM responses is going to get better and better, quite possibly to the point where it is indistinguishable from a human. But it wont matter, because some humans will redefine intelligence to be something else so their place at the top of the intelligence pyramid remains in place.
  - all2 3y ago
    You can coax LLM's into reasoning. I recall someone on here posting a link to prompts that instruct the LLM to reason through a request, and this improves its output significantly. I think what our models are missing is recursion on themselves. You and I can be self-referential, and we are capable of meta thought. We are also capable of "internal dialogue" where we speak and reason internally. The LLMs at present lack even state or memory. Arguably those aren't necessary for "reasoning" capabilities. I wonder how far off "stochastic parrot" is from what we do naturally. I have an image in my head of how we think using associated words/concepts/pictures for learning and I can't imagine it is too different from statistically associated concepts/words. --- This is sort of scatterbrained, and I apologize. I don't have enough time to write a more concise response.
kahnclusions 3y ago
That doesn’t mean it does any “reasoning”. It generates a response text that looks like a response to the input text. The whole point is that it generates responses that aren’t in the training set. But the fact that the response is actually a correct one is just coincidence.
- a_wild_dandan 3y ago
  So when a model succeeds on a reasoning task it's not positive evidence, but when it fails it's negative evidence? The good ol' confirmation bias feedback loop!
a_wild_dandan 3y ago
It’s also unclear that LLMs have no “reasoning loop”, or that a “loop” abstraction is necessary for all reasoning, or that eyeballing wrong answers is a sufficient metric to categorically dismiss “reasoning." A "reasoning loop" argument is especially odd when applied to LLMs...which explicitly have architectural loops, and their output generation is a literal loop. Folks, we don't understand our own minds. You think we already understand a potential alien mind? With our n = 1 examples of generally intelligent mind architectures (ours)? Fat chance. Anyone confidently claiming sweeping, nebulous conclusions about these new models is likely revealing more about their biases than the model's inner workings. We just don't know much. Hot takes on ML are just anthropocentrism Rorschach tests.
- all2 3y ago
  > A "reasoning loop" argument is especially odd when applied to LLMs...which explicitly have architectural loops Oh, I understood that the current crop of LLMs didn't have a way to push data back into itself. I know so little about LM architecture at this point. I need to work my way through the freeAI course that's out there. Maybe a tutorial or two on building one from scratch.

These models produce correct answers to many problems that require “reasoning” (for any sensible meaning of the word), that are not in their training set.

by abm53 3y ago

These models produce correct answers to many problems that require “reasoning” (for any sensible meaning of the word), that are not in their training set.

all2 3y ago

That just shows that parroting will solve a number of problems that require reasoning, that is "reasoning" as we think of it can be reduced to a statistical process. Kids parrot their parents before they understand the meaning of what they are doing or saying. Meaning arises from a much more complex process than seeing/repeating. I think the same will be true with LLMs. True reasoning capabilities will be another revolution entirely. --- This just occurred to me; LLM's have a cargo cult level understanding of anything they've been trained on. Correct answers are actually statistical flukes -- purposeful because that's how we trained the models -- but not actually significant in terms of reasoning.

a_wild_dandan 3y ago
How do we test this alleged distinction between "true reasoning" and statistical parroting? What experiments can we perform on SotA models to make this idea falsifiable? Commonplace hypotheses like these give me strong "No True Scotsman" vibes, but I'm often wrong. Let's agree on a method, and I'll test it.
- Art9681 3y ago
  There is none. No one understands what intelligence really means or self-awareness. Which is why they feel threatened by anything that challenges their self proclaimed uniquely human trait. The goal posts WILL move a lot more before this debate is settled. Inevitably the success rate and accuracy of LLM responses is going to get better and better, quite possibly to the point where it is indistinguishable from a human. But it wont matter, because some humans will redefine intelligence to be something else so their place at the top of the intelligence pyramid remains in place.
- all2 3y ago
  You can coax LLM's into reasoning. I recall someone on here posting a link to prompts that instruct the LLM to reason through a request, and this improves its output significantly. I think what our models are missing is recursion on themselves. You and I can be self-referential, and we are capable of meta thought. We are also capable of "internal dialogue" where we speak and reason internally. The LLMs at present lack even state or memory. Arguably those aren't necessary for "reasoning" capabilities. I wonder how far off "stochastic parrot" is from what we do naturally. I have an image in my head of how we think using associated words/concepts/pictures for learning and I can't imagine it is too different from statistically associated concepts/words. --- This is sort of scatterbrained, and I apologize. I don't have enough time to write a more concise response.

kahnclusions 3y ago

That doesn’t mean it does any “reasoning”. It generates a response text that looks like a response to the input text. The whole point is that it generates responses that aren’t in the training set. But the fact that the response is actually a correct one is just coincidence.

a_wild_dandan 3y ago
So when a model succeeds on a reasoning task it's not positive evidence, but when it fails it's negative evidence? The good ol' confirmation bias feedback loop!

a_wild_dandan 3y ago

It’s also unclear that LLMs have no “reasoning loop”, or that a “loop” abstraction is necessary for all reasoning, or that eyeballing wrong answers is a sufficient metric to categorically dismiss “reasoning." A "reasoning loop" argument is especially odd when applied to LLMs...which explicitly have architectural loops, and their output generation is a literal loop. Folks, we don't understand our own minds. You think we already understand a potential alien mind? With our n = 1 examples of generally intelligent mind architectures (ours)? Fat chance. Anyone confidently claiming sweeping, nebulous conclusions about these new models is likely revealing more about their biases than the model's inner workings. We just don't know much. Hot takes on ML are just anthropocentrism Rorschach tests.

all2 3y ago
> A "reasoning loop" argument is especially odd when applied to LLMs...which explicitly have architectural loops Oh, I understood that the current crop of LLMs didn't have a way to push data back into itself. I know so little about LM architecture at this point. I need to work my way through the freeAI course that's out there. Maybe a tutorial or two on building one from scratch.