5 ms·

> the spicy autocomplete can solve difficult open math problems No it can't. It can't even solve my son's 4th grade math homework. (This is a real use case for

> the spicy autocomplete can solve difficult open math problems No it can't. It can't even solve my son's 4th grade math homework. (This is a real use case for me, not a dumb benchmark.) You just know nothing about math and are happy to parrot bullshit AI salesmen are selling you.

simonw 4mo ago
Reasoning models with access to Python have been able to solve 4th grade math homework for over a year now. Prove me wrong: show me a 4th grade math problem they can't handle.
- otabdeveloper4 4mo ago
  > show me a 4th grade math problem they can't handle Sure. "8 7 6 5 4 3 2 1 - add minus signs and parenthesis to get 31." P.S. There is an answer online and some LLMs will just copy it verbatim. This doesn't count.
  - simonw 4mo ago
    Whoa, 4th grade math problems got hard! I'm not sure how I'd tackle that one myself.
    - simonw 4mo ago
      GPT-5.5 found a solution only after assuming that you're allowed to concatenate numbers together e.g. 8 7 becomes 87 (it complained at first that it was "under-specified") - using Python it brute-forced a solution (actually finding 13): https://chatgpt.com/share/6a1db54f-7ab8-8333-9218-86a469c284bc https://chatgpt.com/share/6a1db54f-7ab8-8333-9218-86a469c284... Are you sure this is 4th grade level?
      - minimaxir 4mo ago
        I questioned OP's "there is an answer online" claim so I checked and the only source found for the original question was a 5th grade Russian school for mathematics. https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221kj6r9fOIshFmCwcNpQDcfEFnFdYezjkm%22%5D,%22action%22:%22open%22,%22userId%22:%22110194568010056725318%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%...
      - MichaelNolan 4mo ago
        Apparently there is a way to solve this without brute forcing all the combinations. It has to do with looking at how many even an odd numbers there are, and taking into account the goal number is odd. And then thinking through the combinations [even-even=even, even-odd=odd,…] Though this is obviously not something I would expect a 4th grader to solve.
  - dwohnitmok 4mo ago
    [dead]
  - DiogenesKynikos 4mo ago
    > 4th grade math problem And it turns out to be an extremely difficult problem given to Russian math prodigies, which requires one to bend the rules and turn "8 7" into "87".
    - otabdeveloper4 4mo ago
      It's a standard "Russian math" problem. There's boatloads more where that came from, and none of them are solved by LLMs.
  - sanderjd 4mo ago
    It's very funny how you chose an example that is both not 4th grade level math and also something the frontier LLMs are much more likely to be able to solve than nearly any 4th grader. This is a counterexample to your argument, not evidence for your claim. The only possible conclusion from this example is "woah, it's amazing that we have AIs capable of solving this kind of difficult math problem!", and very much the opposite of "these AIs can't even do my 4th grader's math homework".
- tomjakubowski 4mo ago
  The images you can't see in the chats are the question sheet from here, which was the first fourth grade math homework assignment I tried. https://www.k5learning.com/worksheets/math/data-graphing/grade-4-comparing-pictographs-a.pdf https://www.k5learning.com/worksheets/math/data-graphing/gra... Fourth graders typically don't have access to Python for their homework assignments. To be fair to the kids, I tried it first without Python: Opus 4.6 (Feb 2026) with default Medium effort. https://claude.ai/share/1533a3e4-6757-4614-b95d-0743350a6598 https://claude.ai/share/1533a3e4-6757-4614-b95d-0743350a6598 pastebin of the reasoning section (no Python): https://pastebin.com/zZeG5ZnJ https://pastebin.com/zZeG5ZnJ It got questions 2 (Shop D) and 5 (280) wrong. It got question 3 right but the work it showed has the numbers for each shop wrong. My fourth grade teacher would have taken off points for that (shout out Mrs. Van Bladel). Here it is again with a prompted nudge to use Python: https://claude.ai/share/e1265efb-0988-40ac-90ac-c76225b67e98 https://claude.ai/share/e1265efb-0988-40ac-90ac-c76225b67e98 pastebin of the reasoning section (with Python): https://pastebin.com/KsP0xxZL https://pastebin.com/KsP0xxZL This time it used Python to "check its work", and answered the same questions incorrectly (2 and 5). To the model's credit, it did show the correct work on answer 3 this time.
  - deleted 4mo ago
    [deleted]
  - simonw 4mo ago
    That's more of a test of vision LLM ability to correctly identify and count things in an image than it is of mathematical reasoning. If you look at the working of your non-Python example it gets most of the counts wrong - identifying shop A as two full notebooks plus one half notebook when it's actually three full notebooks, for example. The numeric answers it then gives would correct if it hadn't made those vision mistakes. I've been testing vision LLMs on counting the number of pelicans in a photo for a while, they're very unreliable at that. The best I've seen is Google Gemini 2.5 if you have it output image segmentation masks (a feature they have not included in the Gemini 3 series yet): https://simonwillison.net/2025/Apr/18/gemini-image-segmentation/ https://simonwillison.net/2025/Apr/18/gemini-image-segmentat... - but that requires additional harness engineering, you need to explicitly cause it to use its image segmentation mechanism.
skinner_ 4mo ago
> You just know nothing about math and are happy to parrot bullshit AI salesmen are selling you. Not the parent poster here. I do know things about math. I wrote a few papers related to the unit distance problem (https://arxiv.org/abs/2311.10069 https://arxiv.org/abs/2311.10069, https://arxiv.org/abs/2406.15317 https://arxiv.org/abs/2406.15317) and spent quite some time trying to solve it. I had no chance of coming up with the proof that the spicy autocomplete came up with. Dumb benchmark, sure.
- otabdeveloper4 4mo ago
  LLMs are good with symbolic manipulation but can't reason. You can skirt around not reasoning in research math because so much of it is just extremely tedious symbolic manipulation. You can't cheat with advanced fourth grade math, though. They don't know algebra yet and can't substitute verbosity for reasoning.
  - skinner_ 4mo ago
    > You can skirt around not reasoning in research math because so much of it is just extremely tedious symbolic manipulation. LOL
threatofrain 4mo ago
We've already long past that threshold.
sanderjd 4mo ago
I would genuinely be interested in knowing what you're doing that led you to this conclusion. I would be shocked if I was unable to solve 4th grade math homework with any of the contemporary frontier models. I spend most days using them to do significantly more complex things than that.
- margalabargala 4mo ago
  If they took a blurry photo of the piece of paper and uploaded to chatGPT saying "solve this" then I would totally believe it. The frontier models are mostly obnoxiously bad at OCR and properly ingesting what's on an image of a page. If you write out the 4th grade math problem, they would have no trouble.
  - otabdeveloper4 4mo ago
    No, LLMs just can't do math.
    - minimaxir 4mo ago
      If your math does not involve multiplying 20 digit numbers, modern LLMs can "do" math even without a Python tool despite the counterintuition of next token prediction.
      - DiogenesKynikos 4mo ago
        And if you give your LLM access to a calculator, it will have to problem multiplying 20-digit numbers.
    - bdamm 4mo ago
      They can definitely recognize the problem class and build programs to do math. So what's the difference? It's like saying that people can't turn high torque nuts on machine bolts, because you can't use your fingers to do it. But you can use a wrench, so effectively, we can turn high torque nuts on machine bolts even though it isn't something we can natively do unaided.
ConceptJunkie 4mo ago
Terrence Tao disagrees with what you're saying. I think he's in a slightly better position to speak on the subject.
- otabdeveloper4 4mo ago
  Terrence Tao is an expert on 4th grade homework? TIL!

simonw 4mo ago

Reasoning models with access to Python have been able to solve 4th grade math homework for over a year now. Prove me wrong: show me a 4th grade math problem they can't handle.

otabdeveloper4 4mo ago
> show me a 4th grade math problem they can't handle Sure. "8 7 6 5 4 3 2 1 - add minus signs and parenthesis to get 31." P.S. There is an answer online and some LLMs will just copy it verbatim. This doesn't count.
- simonw 4mo ago
  Whoa, 4th grade math problems got hard! I'm not sure how I'd tackle that one myself.
  - simonw 4mo ago
    GPT-5.5 found a solution only after assuming that you're allowed to concatenate numbers together e.g. 8 7 becomes 87 (it complained at first that it was "under-specified") - using Python it brute-forced a solution (actually finding 13): https://chatgpt.com/share/6a1db54f-7ab8-8333-9218-86a469c284bc https://chatgpt.com/share/6a1db54f-7ab8-8333-9218-86a469c284... Are you sure this is 4th grade level?
    - minimaxir 4mo ago
      I questioned OP's "there is an answer online" claim so I checked and the only source found for the original question was a 5th grade Russian school for mathematics. https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221kj6r9fOIshFmCwcNpQDcfEFnFdYezjkm%22%5D,%22action%22:%22open%22,%22userId%22:%22110194568010056725318%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%...
    - MichaelNolan 4mo ago
      Apparently there is a way to solve this without brute forcing all the combinations. It has to do with looking at how many even an odd numbers there are, and taking into account the goal number is odd. And then thinking through the combinations [even-even=even, even-odd=odd,…] Though this is obviously not something I would expect a 4th grader to solve.
- dwohnitmok 4mo ago
  [dead]
- DiogenesKynikos 4mo ago
  > 4th grade math problem And it turns out to be an extremely difficult problem given to Russian math prodigies, which requires one to bend the rules and turn "8 7" into "87".
  - otabdeveloper4 4mo ago
    It's a standard "Russian math" problem. There's boatloads more where that came from, and none of them are solved by LLMs.
- sanderjd 4mo ago
  It's very funny how you chose an example that is both not 4th grade level math and also something the frontier LLMs are much more likely to be able to solve than nearly any 4th grader. This is a counterexample to your argument, not evidence for your claim. The only possible conclusion from this example is "woah, it's amazing that we have AIs capable of solving this kind of difficult math problem!", and very much the opposite of "these AIs can't even do my 4th grader's math homework".
tomjakubowski 4mo ago
The images you can't see in the chats are the question sheet from here, which was the first fourth grade math homework assignment I tried. https://www.k5learning.com/worksheets/math/data-graphing/grade-4-comparing-pictographs-a.pdf https://www.k5learning.com/worksheets/math/data-graphing/gra... Fourth graders typically don't have access to Python for their homework assignments. To be fair to the kids, I tried it first without Python: Opus 4.6 (Feb 2026) with default Medium effort. https://claude.ai/share/1533a3e4-6757-4614-b95d-0743350a6598 https://claude.ai/share/1533a3e4-6757-4614-b95d-0743350a6598 pastebin of the reasoning section (no Python): https://pastebin.com/zZeG5ZnJ https://pastebin.com/zZeG5ZnJ It got questions 2 (Shop D) and 5 (280) wrong. It got question 3 right but the work it showed has the numbers for each shop wrong. My fourth grade teacher would have taken off points for that (shout out Mrs. Van Bladel). Here it is again with a prompted nudge to use Python: https://claude.ai/share/e1265efb-0988-40ac-90ac-c76225b67e98 https://claude.ai/share/e1265efb-0988-40ac-90ac-c76225b67e98 pastebin of the reasoning section (with Python): https://pastebin.com/KsP0xxZL https://pastebin.com/KsP0xxZL This time it used Python to "check its work", and answered the same questions incorrectly (2 and 5). To the model's credit, it did show the correct work on answer 3 this time.
- deleted 4mo ago
  [deleted]
- simonw 4mo ago
  That's more of a test of vision LLM ability to correctly identify and count things in an image than it is of mathematical reasoning. If you look at the working of your non-Python example it gets most of the counts wrong - identifying shop A as two full notebooks plus one half notebook when it's actually three full notebooks, for example. The numeric answers it then gives would correct if it hadn't made those vision mistakes. I've been testing vision LLMs on counting the number of pelicans in a photo for a while, they're very unreliable at that. The best I've seen is Google Gemini 2.5 if you have it output image segmentation masks (a feature they have not included in the Gemini 3 series yet): https://simonwillison.net/2025/Apr/18/gemini-image-segmentation/ https://simonwillison.net/2025/Apr/18/gemini-image-segmentat... - but that requires additional harness engineering, you need to explicitly cause it to use its image segmentation mechanism.

skinner_ 4mo ago

> You just know nothing about math and are happy to parrot bullshit AI salesmen are selling you. Not the parent poster here. I do know things about math. I wrote a few papers related to the unit distance problem (https://arxiv.org/abs/2311.10069 https://arxiv.org/abs/2311.10069, https://arxiv.org/abs/2406.15317 https://arxiv.org/abs/2406.15317) and spent quite some time trying to solve it. I had no chance of coming up with the proof that the spicy autocomplete came up with. Dumb benchmark, sure.

otabdeveloper4 4mo ago
LLMs are good with symbolic manipulation but can't reason. You can skirt around not reasoning in research math because so much of it is just extremely tedious symbolic manipulation. You can't cheat with advanced fourth grade math, though. They don't know algebra yet and can't substitute verbosity for reasoning.
- skinner_ 4mo ago
  > You can skirt around not reasoning in research math because so much of it is just extremely tedious symbolic manipulation. LOL

threatofrain 4mo ago

We've already long past that threshold.

sanderjd 4mo ago

I would genuinely be interested in knowing what you're doing that led you to this conclusion. I would be shocked if I was unable to solve 4th grade math homework with any of the contemporary frontier models. I spend most days using them to do significantly more complex things than that.

margalabargala 4mo ago
If they took a blurry photo of the piece of paper and uploaded to chatGPT saying "solve this" then I would totally believe it. The frontier models are mostly obnoxiously bad at OCR and properly ingesting what's on an image of a page. If you write out the 4th grade math problem, they would have no trouble.
- otabdeveloper4 4mo ago
  No, LLMs just can't do math.
  - minimaxir 4mo ago
    If your math does not involve multiplying 20 digit numbers, modern LLMs can "do" math even without a Python tool despite the counterintuition of next token prediction.
    - DiogenesKynikos 4mo ago
      And if you give your LLM access to a calculator, it will have to problem multiplying 20-digit numbers.
  - bdamm 4mo ago
    They can definitely recognize the problem class and build programs to do math. So what's the difference? It's like saying that people can't turn high torque nuts on machine bolts, because you can't use your fingers to do it. But you can use a wrench, so effectively, we can turn high torque nuts on machine bolts even though it isn't something we can natively do unaided.

ConceptJunkie 4mo ago

Terrence Tao disagrees with what you're saying. I think he's in a slightly better position to speak on the subject.

otabdeveloper4 4mo ago
Terrence Tao is an expert on 4th grade homework? TIL!