5 ms·

i'm very skeptical of this paper. >Basic Arithmetic. Another fundamental failure is that LLMs quickly fail in arithmetic as operands increase (Yuan et al., 202

i'm very skeptical of this paper. >Basic Arithmetic. Another fundamental failure is that LLMs quickly fail in arithmetic as operands increase (Yuan et al., 2023; Testolin, 2024), especially in multiplication. Research shows models rely on superficial pattern-matching rather than arithmetic algorithms, thus struggling notably in middle-digits (Deng et al., 2024). Surprisingly, LLMs fail at simpler tasks (determining the last digit) but succeed in harder ones (first digit identification) (Gambardella et al., 2024). Those fundamental inconsistencies lead to failures for practical tasks like temporal reasoning (Su et al., 2024). This is very misleading and I think flat out wrong. What's the best way to falsify this claim? Edit: I tried falsifying it. https://chatgpt.com/share/6999b72a-3a18-800b-856a-0d5da45b94c9 https://chatgpt.com/share/6999b72a-3a18-800b-856a-0d5da45b94... https://chatgpt.com/share/6999b755-62f4-800b-912e-d015f9afc81d https://chatgpt.com/share/6999b755-62f4-800b-912e-d015f9afc8... I provided really hard 20 digit multiplications without tools. If you looked at the reasoning trace, it does what is normally expected and gets it right. I think this is enough to suggest that the claims made in the paper are not valid and LLMs do reason well. To anyone who would disagree, can you provide a counter example that can't be solved using GPT 5 pro but that a normal student could do without mistakes?

simianwords 9mo ago
>Math Word Problem (MWP) Benchmarks. Certain benchmarks inherently possess richer logical structures that facilitate targeted perturbations. MWPs exemplify this, as their logic can be readily abstracted into reusable templates. Researchers use this property to generate variants by sampling numeric values (Gulati et al., 2024; Qian et al., 2024; Li et al., 2024b) or substituting irrelevant entities (Shi et al., 2023; Mirzadeh et al., 2024). Structural transformations – such as exchanging known and unknown components (Deb et al., 2024; Guo et al., 2024a) or applying small alterations that change the logic needed to solve problems (Huang et al., 2025b) – further highlight deeper robustness limitations. I'm willing to bet this is no longer true as well. We have models that are doing better than humans at IMO.
- otabdeveloper4 9mo ago
  > We have models that are doing better than humans at IMO. Not really. From my brief experience they can guess the final answer but the intermediate justifications and proofs are complete hallucinated bullshit. (Possibly because the final answer is usually some sort of neat and beatiful answer and human evaluators don't care about the final answer anyways, in any olympiad you're graded on the soundness of your reasoning.)
  - simianwords 9mo ago
    what's the best way to falsify it?
    - tveita 9mo ago
      You could start by reading research on the topic instead of disregarding expert opinion based on your own gut feeling E.g. https://www.anthropic.com/research/tracing-thoughts-language-model#mental-math https://www.anthropic.com/research/tracing-thoughts-language...
      - simianwords 9mo ago
        It’s specific on Claude.
    - otabdeveloper4 9mo ago
      Falsify what? The claim that LLM's are good for olympiad problems? I'm just an end user who tried to use these "frontier models" to actually solve real olympiad problems. They're useless.
chickenimprint 9mo ago
It's a well known fact that LLMs struggle with basic arithmetic of large numbers, that's not what they are made for. Most chatbots will just call a python interpreter in the background.
- simianwords 9mo ago
  how do you want to falsify it? can you come up with a test?
  - chickenimprint 9mo ago
    Ask a local AI or a chatbot that allows you to disable tool calling to multiply two large number for example. This is what Mistral outputs: The result of multiplying 63,157,997,633 by 63,114,90,009 is: 3,965,689,999,999,999,999,999 (approximately 3.966 × 10²⁴). That's like 5 orders of magnitude off, the scientific notation doesn't even match the full integer, and the mantissa is also slightly wrong.
    - simianwords 9mo ago
      I don't think finding the worst model and asking it to do things is the best way to go. GPT-5 pro without tools can easily solve your question and much harder ones. Rather: does there exist a model that can perform these calculations reliably is a better way to falsify this claim. Else we can always find the worst model with 1B parameters to falsify any claim made on LLM's.
      - AlexeyBrin 9mo ago
        How do you know GPT-5 does not call a Python interpreter remotely on OpenAI servers when you ask it to do arithmetic ? Your prompt goes to their servers, you have no way to know what happens there. The only way to be sure a model calls no tool is to run it locally and control the network.
        9mo ago
throw310822 9mo ago
Just look at the dates of the cited articles. 2023, 2024: that's prehistory, before thinking models anyway. It's like concluding that humans don't understand arithmetic because they can't multiply large numbers at sight.
- simianwords 9mo ago
  i don't get the point of using that in a paper today
  - throw310822 9mo ago
    I'm not sure what the paper is really about despite the enthusiasm of the LLM haters here. Certainly there isn't something called "LLMs" that stayed reasonably the same in the last 4 years- GPT-2 is an LLM but a finding on it most likely doesn't apply to Opus 4.6. You can't document a failure on a 2024 model and claim "LLMs can't do this".
rybosworld 9mo ago
I see that your prompt includes 'Do not use any tools. If you do, write "I USED A TOOL"' This is not a valid experiment, because GPT models always have access to certain tools and will use them even if you tell them not to. They will fib the chain of thought after the fact to make it look like they didn't use a tool. https://www.anthropic.com/research/alignment-faking https://www.anthropic.com/research/alignment-faking It's also well established that all the frontier models use python for math problems, not just GPT family of models.
- simianwords 9mo ago
  Would it convince you if we use the GPT Pro api and explicitly not allow tool access? Is that enough to falsify?
  - jibal 9mo ago
    It's not falsifiable because it's not false.
    - simianwords 9mo ago
      That’s not falsifiable means
      - jibal 9mo ago
        I know what falsifiable means--you're misusing it and I simply adopted your misuse. A claim is falsifiable or not ... it can't be made falsifiable. The way you're using it is "Can we come up with a test to show that it's false"--no, we can't, because it's not false.
        simianwords 9mo ago
        How do you know it’s not false? If one had to prove that it is false, what would you have to do?

simianwords 9mo ago

>Math Word Problem (MWP) Benchmarks. Certain benchmarks inherently possess richer logical structures that facilitate targeted perturbations. MWPs exemplify this, as their logic can be readily abstracted into reusable templates. Researchers use this property to generate variants by sampling numeric values (Gulati et al., 2024; Qian et al., 2024; Li et al., 2024b) or substituting irrelevant entities (Shi et al., 2023; Mirzadeh et al., 2024). Structural transformations – such as exchanging known and unknown components (Deb et al., 2024; Guo et al., 2024a) or applying small alterations that change the logic needed to solve problems (Huang et al., 2025b) – further highlight deeper robustness limitations. I'm willing to bet this is no longer true as well. We have models that are doing better than humans at IMO.

otabdeveloper4 9mo ago
> We have models that are doing better than humans at IMO. Not really. From my brief experience they can guess the final answer but the intermediate justifications and proofs are complete hallucinated bullshit. (Possibly because the final answer is usually some sort of neat and beatiful answer and human evaluators don't care about the final answer anyways, in any olympiad you're graded on the soundness of your reasoning.)
- simianwords 9mo ago
  what's the best way to falsify it?
  - tveita 9mo ago
    You could start by reading research on the topic instead of disregarding expert opinion based on your own gut feeling E.g. https://www.anthropic.com/research/tracing-thoughts-language-model#mental-math https://www.anthropic.com/research/tracing-thoughts-language...
    - simianwords 9mo ago
      It’s specific on Claude.
  - otabdeveloper4 9mo ago
    Falsify what? The claim that LLM's are good for olympiad problems? I'm just an end user who tried to use these "frontier models" to actually solve real olympiad problems. They're useless.

chickenimprint 9mo ago

It's a well known fact that LLMs struggle with basic arithmetic of large numbers, that's not what they are made for. Most chatbots will just call a python interpreter in the background.

simianwords 9mo ago
how do you want to falsify it? can you come up with a test?
- chickenimprint 9mo ago
  Ask a local AI or a chatbot that allows you to disable tool calling to multiply two large number for example. This is what Mistral outputs: The result of multiplying 63,157,997,633 by 63,114,90,009 is: 3,965,689,999,999,999,999,999 (approximately 3.966 × 10²⁴). That's like 5 orders of magnitude off, the scientific notation doesn't even match the full integer, and the mantissa is also slightly wrong.
  - simianwords 9mo ago
    I don't think finding the worst model and asking it to do things is the best way to go. GPT-5 pro without tools can easily solve your question and much harder ones. Rather: does there exist a model that can perform these calculations reliably is a better way to falsify this claim. Else we can always find the worst model with 1B parameters to falsify any claim made on LLM's.
    - AlexeyBrin 9mo ago
      How do you know GPT-5 does not call a Python interpreter remotely on OpenAI servers when you ask it to do arithmetic ? Your prompt goes to their servers, you have no way to know what happens there. The only way to be sure a model calls no tool is to run it locally and control the network.
      - 9mo ago

throw310822 9mo ago

Just look at the dates of the cited articles. 2023, 2024: that's prehistory, before thinking models anyway. It's like concluding that humans don't understand arithmetic because they can't multiply large numbers at sight.

simianwords 9mo ago
i don't get the point of using that in a paper today
- throw310822 9mo ago
  I'm not sure what the paper is really about despite the enthusiasm of the LLM haters here. Certainly there isn't something called "LLMs" that stayed reasonably the same in the last 4 years- GPT-2 is an LLM but a finding on it most likely doesn't apply to Opus 4.6. You can't document a failure on a 2024 model and claim "LLMs can't do this".

rybosworld 9mo ago

I see that your prompt includes 'Do not use any tools. If you do, write "I USED A TOOL"' This is not a valid experiment, because GPT models always have access to certain tools and will use them even if you tell them not to. They will fib the chain of thought after the fact to make it look like they didn't use a tool. https://www.anthropic.com/research/alignment-faking https://www.anthropic.com/research/alignment-faking It's also well established that all the frontier models use python for math problems, not just GPT family of models.

simianwords 9mo ago
Would it convince you if we use the GPT Pro api and explicitly not allow tool access? Is that enough to falsify?
- jibal 9mo ago
  It's not falsifiable because it's not false.
  - simianwords 9mo ago
    That’s not falsifiable means
    - jibal 9mo ago
      I know what falsifiable means--you're misusing it and I simply adopted your misuse. A claim is falsifiable or not ... it can't be made falsifiable. The way you're using it is "Can we come up with a test to show that it's false"--no, we can't, because it's not false.
      - simianwords 9mo ago
        How do you know it’s not false? If one had to prove that it is false, what would you have to do?