3 ms·

People are going to misinterpret this and overgeneralize the claim. This does not say that AI isn't reliable for things. It provides a method for quantifying th

People are going to misinterpret this and overgeneralize the claim. This does not say that AI isn't reliable for things. It provides a method for quantifying the reliability for specific tasks. You wouldn't say that a human who doesn't know how to read isn't reliable in everything, just in reading. Counting is something that even humans need to learn how to do. Toddlers also don't understand quantity. If a 2 year old is able to count to even 10 it's through memorization and not understanding. It takes them like 2 more years of learning before they're able to comprehend things like numerical correspondence. But they do still know how to do other things that aren't counting before then.

irishcoffee 7mo ago
> Counting is something that even humans need to learn how to do. Toddlers also don't understand quantity. If they're able to count to even 10 it's through memorization and not understanding. I completely agree with you. LLMs are regurgitation machines with less intellect than a toddler, you nailed it. AI is here!
coldtea 7mo ago
>Counting is something that even humans need to learn how to do No human who can program, solve advanced math problems, or can talk about advanced problem domains at expert level, however, would fail to count to 5. This is not a mere "LLMs, like humans, also need to be taught this" but points to a fundamental mismatch about how humans and LLMs learn. (And even if they merely needed to be taught, why would their huge corpus fail to cover that "teaching", but cover way more advanced topics in math solving and other domains?)
nkrisc 7mo ago
You’re conflating counting and language. Many animals can count. Counting is recognizing that the box with 3 apples is preferable to the one with 2 apples. Yes, 2 year olds might struggle with the externalization of numeric identities but if you have 1 M&M in one hand and 5 in the other and ask which they want, they’ll take the 5. LLMs have the language part down, but fundamentally can’t count.
- BugsJustFindMe 7mo ago
  The concept of bigger/smaller is useful but is a distinct skill from counting. If you spread the M&Ms apart enough that the part of the brain responsible for gestalt clustering can't group them into a "bigger whole" signal, they'll no longer be able to do the thing you're saying (this is the law of proximity in gestalt psychology).
  - adrian_b 7mo ago
    Most animals can distinguish bigger from smaller. However many animals can distinguish independently small numbers, like 3 or 5, and recognize them whenever they see them. So in this respect, there is little difference between humans and many animals. Humans learn to count to arbitrarily big numbers, but they can still easily recognize only small numbers.
    - BugsJustFindMe 7mo ago
      > many animals can distinguish independently small numbers, like 3 or 5 This is called subitizing. It's distinct from counting. We can see the difference in humans with Simultanagnosia, who are unable to count beyond the subitizing range. Subitizing is categorizing the scale of a small gestalt group. The only thing I've ever seen where an animal appeared to demonstrate counting (up to 3) without training was in rhesus monkeys (maybe also chimpanzees?), but even that experiment could be explained through temporal gestalt. (It's the only reason I know of for them to not have been able to go higher than 3 in that experiment in the context of many other things that they can do.)
      - somethingweird 7mo ago
        Even parrots can count to 6 and more, I would be surprised if primates couldn't.
        BugsJustFindMe 7mo ago
        At least one has maybe been shown to be able to do that with 30 years of focused training, but none have been shown to be able without training. Wild parrots have only demonstrated subitizing and size discrimination, not counting. The overeager do quite often confuse subitizing and size discrimination for counting, though. That's its own problem.
Topfi 7mo ago
Respectfully, toddlers cannot output useable code or have otherwise memorised results to an immense number of maths equations. What this points at is the abstraction/emergence crux of it all. Why does an otherwise very capable LLM such as the GPT-5 series, despite having been trained on vastly more examples of frontend code of all shapes, sizes and quality levels, struggle to abstract all that training data to the point where outputting any frontend that deviates from the clearly used examples? If LLMs, as they are now, were comparable with human learning, there'd be no scenario where a model that can provide output solving highly advanced equations can not count properly. Similarly, a model such as GPT-5 trained on nearly all frontend code ever committed to any repo online, would have internalised more than that one template OpenAI predominantly leaned on. These models, I think at this point there is little doubt, are impressive tools, but they still do not generalise or abstract information in the way a human mind does. Doesn't make them less impactful for industries, etc. but it makes any comparison to humans not very suitable.
- BugsJustFindMe 7mo ago
  > What this points at is the abstraction/emergence crux of it all. Why does This paper has nothing to do with any questions starting with "why". It provides a metric for quantifying error on specific tasks. > If LLMs, as they are now, were comparable with human learning I think I missed the part where they need to be. > struggle to abstract all that training data to the point where outputting any frontend that deviates from the clearly used examples? ... a model such as GPT-5 trained on nearly all frontend code ever committed to any repo online, would have internalised more than that one template OpenAI predominantly leaned on There is a very big and very important difference between producing the same thing again and not being able to produce something else. When not given any reason to produce something else, humans also generate the same thing over and over. That's a problem of missing constraints, not of missing ability. Long before AI there was this thing called Twitter Bootstrap. It dominated the web for...much longer than it should have. And that tragedy was done entirely by us meatsacks (not me personally). Where there's no goal for different output there's no reason to produce different output, and LLMs don't have their own goals because they don't have any mechanisms for desire (we hope). [I've edited this comment for content and format]

irishcoffee 7mo ago

> Counting is something that even humans need to learn how to do. Toddlers also don't understand quantity. If they're able to count to even 10 it's through memorization and not understanding. I completely agree with you. LLMs are regurgitation machines with less intellect than a toddler, you nailed it. AI is here!

coldtea 7mo ago

>Counting is something that even humans need to learn how to do No human who can program, solve advanced math problems, or can talk about advanced problem domains at expert level, however, would fail to count to 5. This is not a mere "LLMs, like humans, also need to be taught this" but points to a fundamental mismatch about how humans and LLMs learn. (And even if they merely needed to be taught, why would their huge corpus fail to cover that "teaching", but cover way more advanced topics in math solving and other domains?)

nkrisc 7mo ago

You’re conflating counting and language. Many animals can count. Counting is recognizing that the box with 3 apples is preferable to the one with 2 apples. Yes, 2 year olds might struggle with the externalization of numeric identities but if you have 1 M&M in one hand and 5 in the other and ask which they want, they’ll take the 5. LLMs have the language part down, but fundamentally can’t count.

BugsJustFindMe 7mo ago
The concept of bigger/smaller is useful but is a distinct skill from counting. If you spread the M&Ms apart enough that the part of the brain responsible for gestalt clustering can't group them into a "bigger whole" signal, they'll no longer be able to do the thing you're saying (this is the law of proximity in gestalt psychology).
- adrian_b 7mo ago
  Most animals can distinguish bigger from smaller. However many animals can distinguish independently small numbers, like 3 or 5, and recognize them whenever they see them. So in this respect, there is little difference between humans and many animals. Humans learn to count to arbitrarily big numbers, but they can still easily recognize only small numbers.
  - BugsJustFindMe 7mo ago
    > many animals can distinguish independently small numbers, like 3 or 5 This is called subitizing. It's distinct from counting. We can see the difference in humans with Simultanagnosia, who are unable to count beyond the subitizing range. Subitizing is categorizing the scale of a small gestalt group. The only thing I've ever seen where an animal appeared to demonstrate counting (up to 3) without training was in rhesus monkeys (maybe also chimpanzees?), but even that experiment could be explained through temporal gestalt. (It's the only reason I know of for them to not have been able to go higher than 3 in that experiment in the context of many other things that they can do.)
    - somethingweird 7mo ago
      Even parrots can count to 6 and more, I would be surprised if primates couldn't.
      - BugsJustFindMe 7mo ago
        At least one has maybe been shown to be able to do that with 30 years of focused training, but none have been shown to be able without training. Wild parrots have only demonstrated subitizing and size discrimination, not counting. The overeager do quite often confuse subitizing and size discrimination for counting, though. That's its own problem.

Topfi 7mo ago

Respectfully, toddlers cannot output useable code or have otherwise memorised results to an immense number of maths equations. What this points at is the abstraction/emergence crux of it all. Why does an otherwise very capable LLM such as the GPT-5 series, despite having been trained on vastly more examples of frontend code of all shapes, sizes and quality levels, struggle to abstract all that training data to the point where outputting any frontend that deviates from the clearly used examples? If LLMs, as they are now, were comparable with human learning, there'd be no scenario where a model that can provide output solving highly advanced equations can not count properly. Similarly, a model such as GPT-5 trained on nearly all frontend code ever committed to any repo online, would have internalised more than that one template OpenAI predominantly leaned on. These models, I think at this point there is little doubt, are impressive tools, but they still do not generalise or abstract information in the way a human mind does. Doesn't make them less impactful for industries, etc. but it makes any comparison to humans not very suitable.

BugsJustFindMe 7mo ago
> What this points at is the abstraction/emergence crux of it all. Why does This paper has nothing to do with any questions starting with "why". It provides a metric for quantifying error on specific tasks. > If LLMs, as they are now, were comparable with human learning I think I missed the part where they need to be. > struggle to abstract all that training data to the point where outputting any frontend that deviates from the clearly used examples? ... a model such as GPT-5 trained on nearly all frontend code ever committed to any repo online, would have internalised more than that one template OpenAI predominantly leaned on There is a very big and very important difference between producing the same thing again and not being able to produce something else. When not given any reason to produce something else, humans also generate the same thing over and over. That's a problem of missing constraints, not of missing ability. Long before AI there was this thing called Twitter Bootstrap. It dominated the web for...much longer than it should have. And that tragedy was done entirely by us meatsacks (not me personally). Where there's no goal for different output there's no reason to produce different output, and LLMs don't have their own goals because they don't have any mechanisms for desire (we hope). [I've edited this comment for content and format]