5 ms·

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting

>On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies surprised this gets voted up given the surprising amount of users on HN who think LLMs can't reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. Last time I was talking about LLM intelligence someone rudely told me to read up on how LLMs work and that we already know exactly how they work and they're just token predictors.

ben_w 2y ago
The loudest people seem to be those with the most extreme positions, and that includes on "is ${specific AI} (useless|superhuman) for ${domain}?". Perhaps it's just perception, but perhaps the arguments make them persist, as CGP Grey pointed out: https://www.youtube.com/watch?v=rE3j_RHkqJc https://www.youtube.com/watch?v=rE3j_RHkqJc As I'm in the middle, I get flack from people on both extremes, as I'm outside their (equivalent of or just literally?) Overton window on this subject. Seems like an odd zone to be in for the opinion "this is a useful tool, but I see loads of ways it can go wrong". Makes me wonder what the real common discourse was of looms during the industrial revolution, and not just the modern summary of that era.
- tkgally 2y ago
  > Makes me wonder what the real common discourse was of looms during the industrial revolution, and not just the modern summary of that era. Interesting question. I did a little searching with help from Claude, ChatGPT, Google, and the Internet Archive. Here are some links to writings from that time: “Thoughts on the use of machines, in the cotton manufacture” (1780) https://archive.org/details/bim_eighteenth-century_thoughts-on-the-use-of-m_1780 https://archive.org/details/bim_eighteenth-century_thoughts-... Excerpt: “How many writers and copiers of books were thrown out of employment, or obliged to change it, by the introduction of printing presses? About ten years ago, when the Spinning Jennies came up, old persons, children, and those who could not easily learn to use the new machines, did suffer, for a while; till families had learned to play into one another's hands, by each taking a different kind of work. But the general benefit, which was received from the machines, very soon silenced all objections. And every sensible man now looks upon them with gratitude and approbation. It is probable, this will be the case in all new inventions.” Kevin Binfield, ed., Writings of the Luddites (1811-1815; 2004) https://ia903409.us.archive.org/16/items/writings-of-the-luddites/Writings%20of%20the%20Luddites.pdf https://ia903409.us.archive.org/16/items/writings-of-the-lud... Robert Owen, Observations on the effect of the manufacturing system (1817) https://archive.org/details/observationsonef00owenrich/page/n5/mode/2up https://archive.org/details/observationsonef00owenrich/page/... William Radcliffe, Origin of the new system of manufacture commonly called power-loom weaving (1828) https://archive.org/details/originofnewsyste0000radc https://archive.org/details/originofnewsyste0000radc “An address to the Glasgow cotton-spinners on the moral bearing of their association” (1838) https://catalogue.nla.gov.au/catalog/6023196 https://catalogue.nla.gov.au/catalog/6023196
vundercind 2y ago
The “surprising gaps” are precisely because they’re not reasoning—or, at least, not “reasoning” about the things a human would be to solve the problems, but about some often-correlated but different set of facts about relationships between tokens in writing. It’s the failure modes that make the distinction clearest. LLM output is only meaningful, in the way we usually mean that, at the point we assigned external, human meaning to it, after the fact. The LLM wouldn’t stop operating or become “confused” if fed gibberish, because the meaning it’s extracting doesn’t depend on the meaning humans assign things, except by coincidence—which coincidence we foster by feeding them with things we do not regard as gibberish, but that’s beside the point so far as how they “really work” goes.
- ninetyninenine 2y ago
  But you also conveniently ignore the success modes where the answer is too novel to be anything other than reasoning. The op clearly said LLMs reason so your opinion is totally against and opposed to the opinion of every author of that academic paper. Why aren’t you condemning this paper?
- naasking 2y ago
  > or, at least, not “reasoning” about the things a human would be to solve the problems You can't actually infer that either. Humans have considerable context that LLMs lack. You have no basis to infer how a human would reason given the same context as an LLM, or vice versa.
  - vundercind 2y ago
    I don't think a human could effectively "reason" after being trained on nonsense (I don't think the training would even take). I think believing generative AI is operating on the same kind of meaning we are is a good way to be surprised when they go from writing like a learned professor for paragraphs to suddenly writing in the same tone and confidence but entirely wrong and with a bunch of made-up crap—it's all made-up crap from their perspective (if you will), we've just guided them into often making up crap that correlates to things we, separately from them, regard (they don't "regard", not in this sense) as non-crap. [EDIT] To put it another way: if these things were trained to, I dunno, generate strings of onomatopoeia animal and environmental noises, I don't think anybody would be confusing what they're doing with anything terribly similar to human cognition, even if the output were structured and followed on from prompts reasonably sensibly and we were able to often find something like meaning or mood or movement or locality in the output—but they'd be doing exactly the same thing they're doing now. I think the form of the output and the training sets we've chosen are what're making people believe they're doing stuff significantly like thinking, but it's all the same to an LLM.
moffkalast 2y ago
The reality is that both can be true at the same time. Yes they're next token predictors, but sometimes the only way to do that correctly is by actually understanding everything that came before and reasoning logically about it. There's some Sutskever quote that if the input to a model is most of a crime novel, and the next token is the name of the perpetrator, then the model understood the novel. Transformers are arbitrary function approximators, so there's no hard limitation on what they can or cannot do.
- ninetyninenine 2y ago
  Perhaps reasoning and understanding are things that don’t exist and that humans themselves are just next token predictors. Heck reasoning and understanding are ill defined concepts. It might even be running on the same bullshit fuel as the word “spirituality”. Maybe the rigorous definition of all these vague words like intelligence or comprehension is really just next token prediction.
  - moffkalast 2y ago
    Eh well they were defined in the world where human intelligence was the only thing around, so there were... baked in assumptions. But as the current definitions go, I don't think there's any need for included consciousness or self awareness for these terms to apply, despite what people seem to immediately jump to. There's no part where it needs to mean human-level proficiency either. Understanding is having knowledge about something, reasoning is making logical conclusions from it, and intelligence is being able to do it continuously with newly presented information. All things that well trained LLMs can arguably do to a detectable extent.
naasking 2y ago
I don't think "next token predictor" and "intelligent" are actually mutually exclusive.
- ninetyninenine 2y ago
  I agree. But then why do so many people take this view point? I mean we all can guess as to why but I want to hear the reasoning from them.

ben_w 2y ago

The loudest people seem to be those with the most extreme positions, and that includes on "is ${specific AI} (useless|superhuman) for ${domain}?". Perhaps it's just perception, but perhaps the arguments make them persist, as CGP Grey pointed out: https://www.youtube.com/watch?v=rE3j_RHkqJc https://www.youtube.com/watch?v=rE3j_RHkqJc As I'm in the middle, I get flack from people on both extremes, as I'm outside their (equivalent of or just literally?) Overton window on this subject. Seems like an odd zone to be in for the opinion "this is a useful tool, but I see loads of ways it can go wrong". Makes me wonder what the real common discourse was of looms during the industrial revolution, and not just the modern summary of that era.

tkgally 2y ago
> Makes me wonder what the real common discourse was of looms during the industrial revolution, and not just the modern summary of that era. Interesting question. I did a little searching with help from Claude, ChatGPT, Google, and the Internet Archive. Here are some links to writings from that time: “Thoughts on the use of machines, in the cotton manufacture” (1780) https://archive.org/details/bim_eighteenth-century_thoughts-on-the-use-of-m_1780 https://archive.org/details/bim_eighteenth-century_thoughts-... Excerpt: “How many writers and copiers of books were thrown out of employment, or obliged to change it, by the introduction of printing presses? About ten years ago, when the Spinning Jennies came up, old persons, children, and those who could not easily learn to use the new machines, did suffer, for a while; till families had learned to play into one another's hands, by each taking a different kind of work. But the general benefit, which was received from the machines, very soon silenced all objections. And every sensible man now looks upon them with gratitude and approbation. It is probable, this will be the case in all new inventions.” Kevin Binfield, ed., Writings of the Luddites (1811-1815; 2004) https://ia903409.us.archive.org/16/items/writings-of-the-luddites/Writings%20of%20the%20Luddites.pdf https://ia903409.us.archive.org/16/items/writings-of-the-lud... Robert Owen, Observations on the effect of the manufacturing system (1817) https://archive.org/details/observationsonef00owenrich/page/n5/mode/2up https://archive.org/details/observationsonef00owenrich/page/... William Radcliffe, Origin of the new system of manufacture commonly called power-loom weaving (1828) https://archive.org/details/originofnewsyste0000radc https://archive.org/details/originofnewsyste0000radc “An address to the Glasgow cotton-spinners on the moral bearing of their association” (1838) https://catalogue.nla.gov.au/catalog/6023196 https://catalogue.nla.gov.au/catalog/6023196

vundercind 2y ago

The “surprising gaps” are precisely because they’re not reasoning—or, at least, not “reasoning” about the things a human would be to solve the problems, but about some often-correlated but different set of facts about relationships between tokens in writing. It’s the failure modes that make the distinction clearest. LLM output is only meaningful, in the way we usually mean that, at the point we assigned external, human meaning to it, after the fact. The LLM wouldn’t stop operating or become “confused” if fed gibberish, because the meaning it’s extracting doesn’t depend on the meaning humans assign things, except by coincidence—which coincidence we foster by feeding them with things we do not regard as gibberish, but that’s beside the point so far as how they “really work” goes.

ninetyninenine 2y ago
But you also conveniently ignore the success modes where the answer is too novel to be anything other than reasoning. The op clearly said LLMs reason so your opinion is totally against and opposed to the opinion of every author of that academic paper. Why aren’t you condemning this paper?
naasking 2y ago
> or, at least, not “reasoning” about the things a human would be to solve the problems You can't actually infer that either. Humans have considerable context that LLMs lack. You have no basis to infer how a human would reason given the same context as an LLM, or vice versa.
- vundercind 2y ago
  I don't think a human could effectively "reason" after being trained on nonsense (I don't think the training would even take). I think believing generative AI is operating on the same kind of meaning we are is a good way to be surprised when they go from writing like a learned professor for paragraphs to suddenly writing in the same tone and confidence but entirely wrong and with a bunch of made-up crap—it's all made-up crap from their perspective (if you will), we've just guided them into often making up crap that correlates to things we, separately from them, regard (they don't "regard", not in this sense) as non-crap. [EDIT] To put it another way: if these things were trained to, I dunno, generate strings of onomatopoeia animal and environmental noises, I don't think anybody would be confusing what they're doing with anything terribly similar to human cognition, even if the output were structured and followed on from prompts reasonably sensibly and we were able to often find something like meaning or mood or movement or locality in the output—but they'd be doing exactly the same thing they're doing now. I think the form of the output and the training sets we've chosen are what're making people believe they're doing stuff significantly like thinking, but it's all the same to an LLM.

moffkalast 2y ago

The reality is that both can be true at the same time. Yes they're next token predictors, but sometimes the only way to do that correctly is by actually understanding everything that came before and reasoning logically about it. There's some Sutskever quote that if the input to a model is most of a crime novel, and the next token is the name of the perpetrator, then the model understood the novel. Transformers are arbitrary function approximators, so there's no hard limitation on what they can or cannot do.

ninetyninenine 2y ago
Perhaps reasoning and understanding are things that don’t exist and that humans themselves are just next token predictors. Heck reasoning and understanding are ill defined concepts. It might even be running on the same bullshit fuel as the word “spirituality”. Maybe the rigorous definition of all these vague words like intelligence or comprehension is really just next token prediction.
- moffkalast 2y ago
  Eh well they were defined in the world where human intelligence was the only thing around, so there were... baked in assumptions. But as the current definitions go, I don't think there's any need for included consciousness or self awareness for these terms to apply, despite what people seem to immediately jump to. There's no part where it needs to mean human-level proficiency either. Understanding is having knowledge about something, reasoning is making logical conclusions from it, and intelligence is being able to do it continuously with newly presented information. All things that well trained LLMs can arguably do to a detectable extent.

naasking 2y ago

I don't think "next token predictor" and "intelligent" are actually mutually exclusive.

ninetyninenine 2y ago
I agree. But then why do so many people take this view point? I mean we all can guess as to why but I want to hear the reasoning from them.