3 ms·

Stop asking LLMs to count! Text is broken into tokens in training (subword/multi-word chunks) rather than individual characters; the model doesn’t truly "see"

Stop asking LLMs to count! Text is broken into tokens in training (subword/multi-word chunks) rather than individual characters; the model doesn’t truly "see" letters or spaces the way humans do. Counting requires exact, step-by-step tracking, but LLMs work probabilistically. It's not much of a help anyway, don't you agree?

jtwoodhouse 1y ago
What does it say about us that we think this is AGI or close to it? Maybe AGI really is here?
FergusArgyll 1y ago
How does reasoning help then?
- mustaphah 1y ago
  IDK. Probably the model's doing some mental gymnastics to figure that out. I was surprised they haven't taught it to count yet. It's a well-known limitation.
  - FergusArgyll 1y ago
    But if tokenization makes them not be able to "see" the letters at all, then no amount of mental gymnastics can save you. I'm aware of the limitation, i'm annoyingly using socratic dialogue to convince you that it is possible to count letters if the model were sufficiently smart.
robryan 1y ago
Seems like a good benchmark for AGI. Start with things that are easy for humans but hard for LLMs currently.
- mustaphah 1y ago
  But they have access to tools (though I'm not sure why they're not using them in this case). Ask it to count using a coding tool, and it will always give you the right answer. Just as humans use tools to overcome their limits, LLMs should do the same.
phyzome 1y ago
Why stop? It's hilarious to watch AI floggers wriggle around trying to explain why AGI is just around the corner but their text-outputting machines can't read text.
camel-cdr 1y ago
How many rs are in a sentence spoken out loud to you? Surely we can't figure it out, because sentences are broken up into syllables when spoken; you don't truly hear individual characters, you hear syllables.
mrkeen 1y ago
Plenty of opportunity to tokenise and re-tokenise here: https://mastodon.social/@kjhealy/114990301650917094 https://mastodon.social/@kjhealy/114990301650917094

jtwoodhouse 1y ago

What does it say about us that we think this is AGI or close to it? Maybe AGI really is here?

FergusArgyll 1y ago

How does reasoning help then?

mustaphah 1y ago
IDK. Probably the model's doing some mental gymnastics to figure that out. I was surprised they haven't taught it to count yet. It's a well-known limitation.
- FergusArgyll 1y ago
  But if tokenization makes them not be able to "see" the letters at all, then no amount of mental gymnastics can save you. I'm aware of the limitation, i'm annoyingly using socratic dialogue to convince you that it is possible to count letters if the model were sufficiently smart.

robryan 1y ago

Seems like a good benchmark for AGI. Start with things that are easy for humans but hard for LLMs currently.

mustaphah 1y ago
But they have access to tools (though I'm not sure why they're not using them in this case). Ask it to count using a coding tool, and it will always give you the right answer. Just as humans use tools to overcome their limits, LLMs should do the same.

phyzome 1y ago

Why stop? It's hilarious to watch AI floggers wriggle around trying to explain why AGI is just around the corner but their text-outputting machines can't read text.

camel-cdr 1y ago

How many rs are in a sentence spoken out loud to you? Surely we can't figure it out, because sentences are broken up into syllables when spoken; you don't truly hear individual characters, you hear syllables.

mrkeen 1y ago

Plenty of opportunity to tokenise and re-tokenise here: https://mastodon.social/@kjhealy/114990301650917094 https://mastodon.social/@kjhealy/114990301650917094