6 ms·

The “surprising gaps” are precisely because they’re not reasoning—or, at least, not “reasoning” about the things a human would be to solve the problems, but abo

The “surprising gaps” are precisely because they’re not reasoning—or, at least, not “reasoning” about the things a human would be to solve the problems, but about some often-correlated but different set of facts about relationships between tokens in writing. It’s the failure modes that make the distinction clearest. LLM output is only meaningful, in the way we usually mean that, at the point we assigned external, human meaning to it, after the fact. The LLM wouldn’t stop operating or become “confused” if fed gibberish, because the meaning it’s extracting doesn’t depend on the meaning humans assign things, except by coincidence—which coincidence we foster by feeding them with things we do not regard as gibberish, but that’s beside the point so far as how they “really work” goes.

ninetyninenine 2y ago
But you also conveniently ignore the success modes where the answer is too novel to be anything other than reasoning. The op clearly said LLMs reason so your opinion is totally against and opposed to the opinion of every author of that academic paper. Why aren’t you condemning this paper?
naasking 2y ago
> or, at least, not “reasoning” about the things a human would be to solve the problems You can't actually infer that either. Humans have considerable context that LLMs lack. You have no basis to infer how a human would reason given the same context as an LLM, or vice versa.
- vundercind 2y ago
  I don't think a human could effectively "reason" after being trained on nonsense (I don't think the training would even take). I think believing generative AI is operating on the same kind of meaning we are is a good way to be surprised when they go from writing like a learned professor for paragraphs to suddenly writing in the same tone and confidence but entirely wrong and with a bunch of made-up crap—it's all made-up crap from their perspective (if you will), we've just guided them into often making up crap that correlates to things we, separately from them, regard (they don't "regard", not in this sense) as non-crap. [EDIT] To put it another way: if these things were trained to, I dunno, generate strings of onomatopoeia animal and environmental noises, I don't think anybody would be confusing what they're doing with anything terribly similar to human cognition, even if the output were structured and followed on from prompts reasonably sensibly and we were able to often find something like meaning or mood or movement or locality in the output—but they'd be doing exactly the same thing they're doing now. I think the form of the output and the training sets we've chosen are what're making people believe they're doing stuff significantly like thinking, but it's all the same to an LLM.
  - ninetyninenine 2y ago
    But how can you be sure? You talk with confidence as if evidence exists to prove what you say but none of this evidence exists. It’s almost as if you’re an LLM yourself making up a claim with zero evidence. Sure you have examples that correlate with your point but nothing that proves your point. Additionally there exists LLM output that runs counter to your point. Explain LLM output that is correct and novel. There exists correct LLM output on queries that are so novel and unique they don’t exist in any form in the training data. You can easily and I mean really easily make an LLM produce such output. Again you’re making up your answer here without proof or evidence which is identical to the extrapolation the LLM does. And your answer runs counter to every academic author on that paper. So what I don’t understand from people like you is the level of unhinged confidence that runs border to religion. Like you were talking about how the wrongness of certain LLM output make the distinction clearest while obviously ignoring the output that makes it unclear. It’s utterly trivial to get LLMs to output things that disprove your point. But what’s more insane is that you can get LLMs to explain all of what’s being debated in this thread to you. https://chatgpt.com/share/674dd1fa-4934-8001-bbda-40fe36907488 https://chatgpt.com/share/674dd1fa-4934-8001-bbda-40fe369074...
    - vundercind 2y ago
      I ignored your other response to me because I didn't see anything in the abstract that contradicted my posts, but maybe there's something deeper in the paper that does. I'll read more of it later. I think, though, the disconnect between us is that I don't see this: > Explain LLM output that is correct and novel. As something I need to do for my position to be strong. It would be if I'd made different claims, but I haven't made those claims. I can see parts of the paper's abstract that would also be relevant and tough to deal with if I'd made those other claims, so I'm guessing those are the parts you think I need to focus on, but I'm not disputing stuff like (paraphrasing) "LLMs may produce output that follows the pattern of a form of reasoned argument they were trained on, not just the particulars" from the abstract. Sure, maybe they do, OK. I don't subscribe to (and don't really understand the motivation for) claims that generative AI can't produce output that's not explicitly in their training set, which is a claim I have seen and (I think?) one you're taking me as promoting. Friggin' Markov chain text generators can, why couldn't LLMs? Another formulation is that everything they output is a result of what they were trained on, which is stronger but only because it's, like, tautologically true and not very interesting.

ninetyninenine 2y ago

But you also conveniently ignore the success modes where the answer is too novel to be anything other than reasoning. The op clearly said LLMs reason so your opinion is totally against and opposed to the opinion of every author of that academic paper. Why aren’t you condemning this paper?

naasking 2y ago

> or, at least, not “reasoning” about the things a human would be to solve the problems You can't actually infer that either. Humans have considerable context that LLMs lack. You have no basis to infer how a human would reason given the same context as an LLM, or vice versa.

vundercind 2y ago
I don't think a human could effectively "reason" after being trained on nonsense (I don't think the training would even take). I think believing generative AI is operating on the same kind of meaning we are is a good way to be surprised when they go from writing like a learned professor for paragraphs to suddenly writing in the same tone and confidence but entirely wrong and with a bunch of made-up crap—it's all made-up crap from their perspective (if you will), we've just guided them into often making up crap that correlates to things we, separately from them, regard (they don't "regard", not in this sense) as non-crap. [EDIT] To put it another way: if these things were trained to, I dunno, generate strings of onomatopoeia animal and environmental noises, I don't think anybody would be confusing what they're doing with anything terribly similar to human cognition, even if the output were structured and followed on from prompts reasonably sensibly and we were able to often find something like meaning or mood or movement or locality in the output—but they'd be doing exactly the same thing they're doing now. I think the form of the output and the training sets we've chosen are what're making people believe they're doing stuff significantly like thinking, but it's all the same to an LLM.
- ninetyninenine 2y ago
  But how can you be sure? You talk with confidence as if evidence exists to prove what you say but none of this evidence exists. It’s almost as if you’re an LLM yourself making up a claim with zero evidence. Sure you have examples that correlate with your point but nothing that proves your point. Additionally there exists LLM output that runs counter to your point. Explain LLM output that is correct and novel. There exists correct LLM output on queries that are so novel and unique they don’t exist in any form in the training data. You can easily and I mean really easily make an LLM produce such output. Again you’re making up your answer here without proof or evidence which is identical to the extrapolation the LLM does. And your answer runs counter to every academic author on that paper. So what I don’t understand from people like you is the level of unhinged confidence that runs border to religion. Like you were talking about how the wrongness of certain LLM output make the distinction clearest while obviously ignoring the output that makes it unclear. It’s utterly trivial to get LLMs to output things that disprove your point. But what’s more insane is that you can get LLMs to explain all of what’s being debated in this thread to you. https://chatgpt.com/share/674dd1fa-4934-8001-bbda-40fe36907488 https://chatgpt.com/share/674dd1fa-4934-8001-bbda-40fe369074...
  - vundercind 2y ago
    I ignored your other response to me because I didn't see anything in the abstract that contradicted my posts, but maybe there's something deeper in the paper that does. I'll read more of it later. I think, though, the disconnect between us is that I don't see this: > Explain LLM output that is correct and novel. As something I need to do for my position to be strong. It would be if I'd made different claims, but I haven't made those claims. I can see parts of the paper's abstract that would also be relevant and tough to deal with if I'd made those other claims, so I'm guessing those are the parts you think I need to focus on, but I'm not disputing stuff like (paraphrasing) "LLMs may produce output that follows the pattern of a form of reasoned argument they were trained on, not just the particulars" from the abstract. Sure, maybe they do, OK. I don't subscribe to (and don't really understand the motivation for) claims that generative AI can't produce output that's not explicitly in their training set, which is a claim I have seen and (I think?) one you're taking me as promoting. Friggin' Markov chain text generators can, why couldn't LLMs? Another formulation is that everything they output is a result of what they were trained on, which is stronger but only because it's, like, tautologically true and not very interesting.