3 ms·

The images you can't see in the chats are the question sheet from here, which was the first fourth grade math homework assignment I tried. https://www.k5learnin

The images you can't see in the chats are the question sheet from here, which was the first fourth grade math homework assignment I tried. https://www.k5learning.com/worksheets/math/data-graphing/grade-4-comparing-pictographs-a.pdf https://www.k5learning.com/worksheets/math/data-graphing/gra... Fourth graders typically don't have access to Python for their homework assignments. To be fair to the kids, I tried it first without Python: Opus 4.6 (Feb 2026) with default Medium effort. https://claude.ai/share/1533a3e4-6757-4614-b95d-0743350a6598 https://claude.ai/share/1533a3e4-6757-4614-b95d-0743350a6598 pastebin of the reasoning section (no Python): https://pastebin.com/zZeG5ZnJ https://pastebin.com/zZeG5ZnJ It got questions 2 (Shop D) and 5 (280) wrong. It got question 3 right but the work it showed has the numbers for each shop wrong. My fourth grade teacher would have taken off points for that (shout out Mrs. Van Bladel). Here it is again with a prompted nudge to use Python: https://claude.ai/share/e1265efb-0988-40ac-90ac-c76225b67e98 https://claude.ai/share/e1265efb-0988-40ac-90ac-c76225b67e98 pastebin of the reasoning section (with Python): https://pastebin.com/KsP0xxZL https://pastebin.com/KsP0xxZL This time it used Python to "check its work", and answered the same questions incorrectly (2 and 5). To the model's credit, it did show the correct work on answer 3 this time.

deleted 5mo ago
[deleted]
simonw 5mo ago
That's more of a test of vision LLM ability to correctly identify and count things in an image than it is of mathematical reasoning. If you look at the working of your non-Python example it gets most of the counts wrong - identifying shop A as two full notebooks plus one half notebook when it's actually three full notebooks, for example. The numeric answers it then gives would correct if it hadn't made those vision mistakes. I've been testing vision LLMs on counting the number of pelicans in a photo for a while, they're very unreliable at that. The best I've seen is Google Gemini 2.5 if you have it output image segmentation masks (a feature they have not included in the Gemini 3 series yet): https://simonwillison.net/2025/Apr/18/gemini-image-segmentation/ https://simonwillison.net/2025/Apr/18/gemini-image-segmentat... - but that requires additional harness engineering, you need to explicitly cause it to use its image segmentation mechanism.
- tomjakubowski 5mo ago
  Fourth grade math's† students are learning geometry and how to draw simple plots. Vision ability (or tactile ability, for visually impaired students) is pretty important to understanding and solving those homework problems. †: think "bo's'n"

deleted 5mo ago

[deleted]

simonw 5mo ago

That's more of a test of vision LLM ability to correctly identify and count things in an image than it is of mathematical reasoning. If you look at the working of your non-Python example it gets most of the counts wrong - identifying shop A as two full notebooks plus one half notebook when it's actually three full notebooks, for example. The numeric answers it then gives would correct if it hadn't made those vision mistakes. I've been testing vision LLMs on counting the number of pelicans in a photo for a while, they're very unreliable at that. The best I've seen is Google Gemini 2.5 if you have it output image segmentation masks (a feature they have not included in the Gemini 3 series yet): https://simonwillison.net/2025/Apr/18/gemini-image-segmentation/ https://simonwillison.net/2025/Apr/18/gemini-image-segmentat... - but that requires additional harness engineering, you need to explicitly cause it to use its image segmentation mechanism.

tomjakubowski 5mo ago
Fourth grade math's† students are learning geometry and how to draw simple plots. Vision ability (or tactile ability, for visually impaired students) is pretty important to understanding and solving those homework problems. †: think "bo's'n"