5 ms·

LLMs are all fake AI. As the recently released Apple study demonstrates, LLMs don't reason, they just pattern match. That's not "intelligence" however you defin

LLMs are all fake AI. As the recently released Apple study demonstrates, LLMs don't reason, they just pattern match. That's not "intelligence" however you define it because they can only solve things that are already within their training set. In this case, it would have been better for the AI industry if it had been 700 programmers, because then the rest of the industry could have argued that the utter trash code Builder.ai generated was the result of human coders spending a few minutes haphazardly typing out random code, and not the result of a specialty-trained LLM.

aeve890 1y ago
>As the recently released Apple study demonstrates, LLMs don't reason, they just pattern match Hold on a minute I was under the impression that "reasoning" was just marketing buzzword the same as "hallucinations", because how tf anyone expected GPUs to "reason" and "hallucinate" when even neurology/psychology don't have a strict definition of those processes.
- jacobr1 1y ago
  No, the definitions are very much up for debate, but there is an actual process here. "Reasoning" in this case means having the model not just produce whatever output is requested directly, but also spend some time writing out its thoughts about how to produce that output. Early version of this were just prompt engineering where you ask the model to produce its "chain of thought" or "work step by step" on how to approach the problem. Later this was trained into the model directly with traces on this intermediate thinking, especially for multistep problems, without the need for explicit prompting. And then architecturally these models now have different ways to determine when to stop "reasoning" to skip to generating actual output. I don't have a strict enough definition to debate if this reasoning is "real" - but from personal experience it certainly appears to be performing something that at least "looks" like inductive thought, and leads to better answers than prior model generations without reasoning/thinking enabled.
  - codr7 1y ago
    Reasoning means what reasoning always meant. Selling an algorithm that can write a list of steps as reasoning is bordering on fraud. It's not uncommon that they guess the right solution, and then "reason" their way out of it.
    - ben_w 1y ago
      > Reasoning means what reasoning always meant. """Reasoning: The deduction of inferences or interpretations from premises.""" Sounds like any logic program to me? Apparently from Latin, ratiō, which has meanings including "explanation" and "computation"?
  - klank 1y ago
    It's gradient descent. Why are we surprised when the answers get better the more we do it? Sometimes you're stuck in a local max/minima, and you hallucinate. Am I oversimplifying it? Is everybody else over-mystifying it?
    - tsimionescu 1y ago
      Gradient descent is how the model weights are adjusted during training. There is no gradient descent, and nothing even remotely similar to it, that happens during inference.
      - klank 1y ago
        Fair, thanks for pointing that out. If you allow me to view the weights of a model as the axioms in an axiomatic system, my (admittedly limited) understanding of modern "AI" inference is that it adds no net new information/knowledge, just more specific expressions of the underlying structure (as defined by the model weights). So while that does undercut my original flippancy of it being "nothing but gradient descent" I don't think it runs counter to my original point that nothing particularly "uncanny" is happening here, no?
        1y ago
throwaway314155 1y ago
> As the recently released Apple study demonstrates, LLMs don't reason Where is everyone getting this misconception? I have seen it several times. First off, the study doesn't even try to qualify whether or not these models use "actual reasoning" - that's outside of the scope. They merely examine how effective thinking/reasoning _is_ at producing better results. They found that - indeed - reasoning improves performance. But the crucial result is that it only improves performance up to a certain difficulty-cliff - at which point thinking makes no discernable difference due to a model collapse of sorts. It's important to read the papers you're using to champion your personal biases.
- miltonlost 1y ago
  An LLM was given the algorithm to Towers of Hanoi and was unable to solve it. There is no "reasoning".
  - throwaway314155 1y ago
    You're conflating two definitions of reasoning. LLM's have struggled with visual reasoning since their inception because guess what!? they're trained on language mostly, not a 3d environment. LLM's aren't magic - those who claim they are are hyping for some reason or another. Ignore them. View AI objectively. Ignore your bias.
    - tsimionescu 1y ago
      Visual reasoning is not required to follow an algorithm that was laid out. The fact that the model can't execute an algorithm it was provided proves that it is not able to do deductive synbolic reasoning in general, since that is all that would have been required required. I would agree with you if this were about inventing the algorithm for itself - it may well be that you'd need some amount of visual reasoning to come up with it. But that's not what the GP (or the paper) were talking about.
      - deleted 1y ago
        [deleted]
UebVar 1y ago
> because they can only solve things that are already within their training set. That is just plain wrong, as anybody who spent more than 10 minutes with a LLM within the last 3 years can attest. Give it a try, especially if you care to have an opinion on them. Ask an absurd question (that can be, in principle, answered) that nobody has asked before and see how it performs generalizing. The hype is real. I'm interested what study you refer to. Because I'm interested in their methods and what they actually found out.
- spion 1y ago
  What you think is an absurd question may not be as absurd as it seems, given the trillions of tokens of data on the internet, including its darkest corners. In my experience, its better to simply try using LLMs in areas where they don't have a lot of training data (e.g. reasoning about the behaviour of terraform plans). Its not a hard cutoff of being _only_ able to reason exactly about solved things, but its not too far off as a first approximation. The researchers took exiting known problems and parameterised their difficulty [1]. While most of these are not by any means easy for humans, the interesting observation to me was that the failure_N was not proportional to the complexity of the problem, but more with how common solution "printouts" for that size of the problem can typically be encountered in the training data. For example, "towers of hanoi" which has printouts of solutions for a variety of sizes went to very large number of steps N, while the river crossing, which is almost entirely not present in the training data for N larger than 3, failed above pretty much that exact number. [1]: https://machinelearning.apple.com/research/illusion-of-thinking https://machinelearning.apple.com/research/illusion-of-think...
  - CSSer 1y ago
    It doesn't help that thanks to RLHF, every time a good example of this gains popularity, e.g. "How many Rs are in 'strawberry'?", it's often snuffed out quickly. If I worked at a company with an LLM product, I'd build tooling to look for these kinds of examples in social media or directly in usage data so they can be prioritized for fixes. I don't know how to feel about this. On the one hand, it's sort of like red teaming. On the other hand, it clearly gives consumers a false sense of ability.
deleted 1y ago
[deleted]
ChrisMarshallNY 1y ago
> because they can only solve things that are already within their training set I just gave up on using SwiftUI for a rewrite of a backend dashboard tool. The LLM didn't give up. It kept suggesting wilder, and less stable ideas, until I realized that this was a rabbithole full of misery, and went back to UIKit. It wasn't the LLM's fault. SwiftUI just isn't ready for the particular functionality I needed, and I guess that a day of watching ChatGPT get more and more desperate, saved me a lot of time. But the LLM didn't give up, which is maybe ot-nay oo-tay ight-bray. https://despair.com/cdn/shop/files/stupidity.jpg https://despair.com/cdn/shop/files/stupidity.jpg
meowface 1y ago
AI skepticism is like a religion at this point. Weird it's so prominent on a tech site. (The Apple paper has had many serious holes poked in it.)
- b00ty4breakfast 1y ago
  Well, if the thing is truly capable of reason, then we have an obligation to put the kibosh on the entire endeavor because we're using a potentially intelligent entity as slave labor. At best, we're re-inventing factory farming and at worst we're re-inventing chattel slavery. Neither of those situations is something I'm personally ok with allowing to continue
  - klank 1y ago
    I concur. I also find the assumption that tech-savvy individuals would inherently be for what we currently call AI to itself, be weird. Unfortunately I feel as though being knowledgable or capable within an area is conflated with an over-acceptance of that area. If anything, the more I've learned about technology, and the more experienced I am, the more fearful and cautious I am with it.
    - b00ty4breakfast 1y ago
      the biggest techno-pessimists I know are all in the tech industry. On the other hand, most of the "blindly accept every new technique or gadget" folks I know are also tech workers so maybe there's nothing going on there. Wonder if there's study on this
  - elcritch 1y ago
    Reason can be distinct from sapience.
    - b00ty4breakfast 1y ago
      can it? the only example we have of something reasoning on this level is humans and we are most definitely sentient despite our best efforts to sometimes appear otherwise. And even if the two could theoretically be separated, how sure are we that these AI agents are in that category? I'm pretty sure they are neither, but that doesn't mean it isn't a pretty abhorrent possibility that should be addressed.
simonw 1y ago
On that basis AI has been "fake AI" since the term AI was first coined in 1956. The "AI isn't really intelligence" argument is so tired now it has a whole Wikipedia page about it: https://en.m.wikipedia.org/wiki/AI_effect https://en.m.wikipedia.org/wiki/AI_effect
- gamblor956 1y ago
  I agree that all AI has been fake AI since the term was first coined. Researchers in the field used to acknowledge that their computational models weren't anywhere close to AI. That all changed when greed became the driving motivation of tech.
ben_w 1y ago
> As the recently released Apple study demonstrates The Apple study that did Towers of Hanoi and concluded that giving up when the answers would have been too long to fit in the output window was a sign of "not reasoning"? https://xcancel.com/scaling01/status/1931783050511126954 https://xcancel.com/scaling01/status/1931783050511126954 I mean, on that basis, anyone who ever went "TL;DR" is also demonstrating that humans don't reason. > That's not "intelligence" however you define it because they can only solve things that are already within their training set. This is proven untrue by, amongst other things, looking at them playing chess. They can and do play moves not found in the training data: https://www.lesswrong.com/posts/yzGDwpRBx6TEcdeA5/a-chess-gpt-linear-emergent-world-representation https://www.lesswrong.com/posts/yzGDwpRBx6TEcdeA5/a-chess-gp...

aeve890 1y ago

>As the recently released Apple study demonstrates, LLMs don't reason, they just pattern match Hold on a minute I was under the impression that "reasoning" was just marketing buzzword the same as "hallucinations", because how tf anyone expected GPUs to "reason" and "hallucinate" when even neurology/psychology don't have a strict definition of those processes.

jacobr1 1y ago
No, the definitions are very much up for debate, but there is an actual process here. "Reasoning" in this case means having the model not just produce whatever output is requested directly, but also spend some time writing out its thoughts about how to produce that output. Early version of this were just prompt engineering where you ask the model to produce its "chain of thought" or "work step by step" on how to approach the problem. Later this was trained into the model directly with traces on this intermediate thinking, especially for multistep problems, without the need for explicit prompting. And then architecturally these models now have different ways to determine when to stop "reasoning" to skip to generating actual output. I don't have a strict enough definition to debate if this reasoning is "real" - but from personal experience it certainly appears to be performing something that at least "looks" like inductive thought, and leads to better answers than prior model generations without reasoning/thinking enabled.
- codr7 1y ago
  Reasoning means what reasoning always meant. Selling an algorithm that can write a list of steps as reasoning is bordering on fraud. It's not uncommon that they guess the right solution, and then "reason" their way out of it.
  - ben_w 1y ago
    > Reasoning means what reasoning always meant. """Reasoning: The deduction of inferences or interpretations from premises.""" Sounds like any logic program to me? Apparently from Latin, ratiō, which has meanings including "explanation" and "computation"?
- klank 1y ago
  It's gradient descent. Why are we surprised when the answers get better the more we do it? Sometimes you're stuck in a local max/minima, and you hallucinate. Am I oversimplifying it? Is everybody else over-mystifying it?
  - tsimionescu 1y ago
    Gradient descent is how the model weights are adjusted during training. There is no gradient descent, and nothing even remotely similar to it, that happens during inference.
    - klank 1y ago
      Fair, thanks for pointing that out. If you allow me to view the weights of a model as the axioms in an axiomatic system, my (admittedly limited) understanding of modern "AI" inference is that it adds no net new information/knowledge, just more specific expressions of the underlying structure (as defined by the model weights). So while that does undercut my original flippancy of it being "nothing but gradient descent" I don't think it runs counter to my original point that nothing particularly "uncanny" is happening here, no?
      - 1y ago

throwaway314155 1y ago

> As the recently released Apple study demonstrates, LLMs don't reason Where is everyone getting this misconception? I have seen it several times. First off, the study doesn't even try to qualify whether or not these models use "actual reasoning" - that's outside of the scope. They merely examine how effective thinking/reasoning _is_ at producing better results. They found that - indeed - reasoning improves performance. But the crucial result is that it only improves performance up to a certain difficulty-cliff - at which point thinking makes no discernable difference due to a model collapse of sorts. It's important to read the papers you're using to champion your personal biases.

miltonlost 1y ago
An LLM was given the algorithm to Towers of Hanoi and was unable to solve it. There is no "reasoning".
- throwaway314155 1y ago
  You're conflating two definitions of reasoning. LLM's have struggled with visual reasoning since their inception because guess what!? they're trained on language mostly, not a 3d environment. LLM's aren't magic - those who claim they are are hyping for some reason or another. Ignore them. View AI objectively. Ignore your bias.
  - tsimionescu 1y ago
    Visual reasoning is not required to follow an algorithm that was laid out. The fact that the model can't execute an algorithm it was provided proves that it is not able to do deductive synbolic reasoning in general, since that is all that would have been required required. I would agree with you if this were about inventing the algorithm for itself - it may well be that you'd need some amount of visual reasoning to come up with it. But that's not what the GP (or the paper) were talking about.
    - deleted 1y ago
      [deleted]

UebVar 1y ago

> because they can only solve things that are already within their training set. That is just plain wrong, as anybody who spent more than 10 minutes with a LLM within the last 3 years can attest. Give it a try, especially if you care to have an opinion on them. Ask an absurd question (that can be, in principle, answered) that nobody has asked before and see how it performs generalizing. The hype is real. I'm interested what study you refer to. Because I'm interested in their methods and what they actually found out.

spion 1y ago
What you think is an absurd question may not be as absurd as it seems, given the trillions of tokens of data on the internet, including its darkest corners. In my experience, its better to simply try using LLMs in areas where they don't have a lot of training data (e.g. reasoning about the behaviour of terraform plans). Its not a hard cutoff of being _only_ able to reason exactly about solved things, but its not too far off as a first approximation. The researchers took exiting known problems and parameterised their difficulty [1]. While most of these are not by any means easy for humans, the interesting observation to me was that the failure_N was not proportional to the complexity of the problem, but more with how common solution "printouts" for that size of the problem can typically be encountered in the training data. For example, "towers of hanoi" which has printouts of solutions for a variety of sizes went to very large number of steps N, while the river crossing, which is almost entirely not present in the training data for N larger than 3, failed above pretty much that exact number. [1]: https://machinelearning.apple.com/research/illusion-of-thinking https://machinelearning.apple.com/research/illusion-of-think...
- CSSer 1y ago
  It doesn't help that thanks to RLHF, every time a good example of this gains popularity, e.g. "How many Rs are in 'strawberry'?", it's often snuffed out quickly. If I worked at a company with an LLM product, I'd build tooling to look for these kinds of examples in social media or directly in usage data so they can be prioritized for fixes. I don't know how to feel about this. On the one hand, it's sort of like red teaming. On the other hand, it clearly gives consumers a false sense of ability.

deleted 1y ago

[deleted]

ChrisMarshallNY 1y ago

> because they can only solve things that are already within their training set I just gave up on using SwiftUI for a rewrite of a backend dashboard tool. The LLM didn't give up. It kept suggesting wilder, and less stable ideas, until I realized that this was a rabbithole full of misery, and went back to UIKit. It wasn't the LLM's fault. SwiftUI just isn't ready for the particular functionality I needed, and I guess that a day of watching ChatGPT get more and more desperate, saved me a lot of time. But the LLM didn't give up, which is maybe ot-nay oo-tay ight-bray. https://despair.com/cdn/shop/files/stupidity.jpg https://despair.com/cdn/shop/files/stupidity.jpg

meowface 1y ago

AI skepticism is like a religion at this point. Weird it's so prominent on a tech site. (The Apple paper has had many serious holes poked in it.)

b00ty4breakfast 1y ago
Well, if the thing is truly capable of reason, then we have an obligation to put the kibosh on the entire endeavor because we're using a potentially intelligent entity as slave labor. At best, we're re-inventing factory farming and at worst we're re-inventing chattel slavery. Neither of those situations is something I'm personally ok with allowing to continue
- klank 1y ago
  I concur. I also find the assumption that tech-savvy individuals would inherently be for what we currently call AI to itself, be weird. Unfortunately I feel as though being knowledgable or capable within an area is conflated with an over-acceptance of that area. If anything, the more I've learned about technology, and the more experienced I am, the more fearful and cautious I am with it.
  - b00ty4breakfast 1y ago
    the biggest techno-pessimists I know are all in the tech industry. On the other hand, most of the "blindly accept every new technique or gadget" folks I know are also tech workers so maybe there's nothing going on there. Wonder if there's study on this
- elcritch 1y ago
  Reason can be distinct from sapience.
  - b00ty4breakfast 1y ago
    can it? the only example we have of something reasoning on this level is humans and we are most definitely sentient despite our best efforts to sometimes appear otherwise. And even if the two could theoretically be separated, how sure are we that these AI agents are in that category? I'm pretty sure they are neither, but that doesn't mean it isn't a pretty abhorrent possibility that should be addressed.

simonw 1y ago

On that basis AI has been "fake AI" since the term AI was first coined in 1956. The "AI isn't really intelligence" argument is so tired now it has a whole Wikipedia page about it: https://en.m.wikipedia.org/wiki/AI_effect https://en.m.wikipedia.org/wiki/AI_effect

gamblor956 1y ago
I agree that all AI has been fake AI since the term was first coined. Researchers in the field used to acknowledge that their computational models weren't anywhere close to AI. That all changed when greed became the driving motivation of tech.

ben_w 1y ago

> As the recently released Apple study demonstrates The Apple study that did Towers of Hanoi and concluded that giving up when the answers would have been too long to fit in the output window was a sign of "not reasoning"? https://xcancel.com/scaling01/status/1931783050511126954 https://xcancel.com/scaling01/status/1931783050511126954 I mean, on that basis, anyone who ever went "TL;DR" is also demonstrating that humans don't reason. > That's not "intelligence" however you define it because they can only solve things that are already within their training set. This is proven untrue by, amongst other things, looking at them playing chess. They can and do play moves not found in the training data: https://www.lesswrong.com/posts/yzGDwpRBx6TEcdeA5/a-chess-gpt-linear-emergent-world-representation https://www.lesswrong.com/posts/yzGDwpRBx6TEcdeA5/a-chess-gp...