8 ms·

Isn't that just an artifact caused by the tokenization of the training and input data? See https://platform.openai.com/tokenizer https://platform.openai.com/t

by amai 1y ago

Isn't that just an artifact caused by the tokenization of the training and input data? See https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer https://github.com/openai/tiktoken https://github.com/openai/tiktoken

falcor84 1y ago
Where in the tokenization does the 3rd b come from?
- IanCal 1y ago
  The tokenisation means they don’t see the letters at all. They see something like this - to convert just some tokens to words How many 538 do you see in 423, 4144, 9890?
  - minimaxir 1y ago
    LLMs don’t see token ids, they see token embeddings that map to those ids, and those embeddings are correlated. The hypothetical embeddings of 538, 423, 4144, and 9890 are likely strongly correlated in the process of training the LLM and the downstream LLM should be able to leverage those patterns to solve the question correctly. Even more so since the training process likely has many examples of similar highly correlated embeddings to identify the next similar token.
    - IanCal 1y ago
      But vitally they are not explicitly shown the letters individually and so “count the letters” is a much harder problem to solve than it is for us.
andrewmcwatters 1y ago
No, it's the entire architecture of the model. There's no real reasoning. It seems that reasoning is just a feedback loop on top of existing autocompletion. It's really disingenuous for the industry to call warming tokens for output, "reasoning," as if some autocomplete before more autocomplete is all we needed to solve the issue of consciousness. Edit: Letter frequency apparently has just become another scripted output, like doing arithmetic. LLMs don't have the ability to do this sort of work inherently, so they're trained to offload the task. Edit: This comment appears to be wildly upvoted and downvoted. If you have anything to add besides reactionary voting, please contribute to the discussion.
- spwa4 1y ago
  I had a fun experience recently. I asked one of my daughters how many r's there are in strawberry. Her answer? Two ... Of course then you ask her to write it and of course things get fixed. But strange.
  - andrewmcwatters 1y ago
    I think that's supposed to be the idea of reasoning functionality, but in practice, it just seems to allow responses to continue longer than that would have otherwise by bisecting the output into warming an output and then using maybe what we would consider cached tokens to assist with further contextual lookups. That is to say, you can obtain the same process by talking to "non-reasoning" models.
  - furyofantares 1y ago
    To be honest, if a kid asked me how many r's in strawberry, I would assume they were asking how many r's at the end and say 2.
  - ziml77 1y ago
    I hate to break it to you but I think your child might actually have gotten swapped in the hospital with an LLM.
- exasperaited 1y ago
  In ten years time an LLM lawyer will lose a legal case for someone who can no longer afford a real lawyer because there are so few left. And it'll be because the layers of bodges in the model caused it to go crazy, insult the judge and threaten to burn down the courthouse. There will be a series of analytical articles in the mainstream press, the tech industry will write it off as a known problem with tokenisation that they can't fix because nobody really writes code anymore. The LLM megacorp will just add a disclaimer: the software should not be used in legal actions concerning fruit companies and they disclaim all losses.
  - Terr_ 1y ago
wongarsu 1y ago
It can spell the word (writing each letter in uppercase followed by a whitespace, which should turn each letter with its whitespace into a separate token). It also has reasoning tokens to use as scratch space, and previous models have demonstrated knowledge of the fact that spelling words is a useful step to counting letters. Tokenization makes the problem difficult, but not solving it is still a reasoning/intelligence issue
- pxc 1y ago
  Here's an example of what gpt-oss-20b (at the default mxfp4 precision) does with this question: > How many "s"es are in the word "Mississippi"? The "thinking portion" is: > Count letters: M i s s i s s i p p i -> s appears 4 times? Actually Mississippi has s's: positions 3,4,6,7 = 4. The answer is: > The word “Mississippi” contains four letter “s” s. They can indeed do some simple pattern matching on the query, separate the letters out into separate tokens, and count them without having to do something like run code in a sandbox and ask it the answer. The issue here is just that this workaround/strategy is only trained into the "thinking" models, afaict.
  - johnecheck 1y ago
    That proves nothing. The fact that Mississippi has 4 "s" is far more likely to be in the training data than the fact that blueberry has 2 "b"s. And now that fact is going to be in the data for the next round of training. We'll need to need to try some other words on the next model.
    - pxc 1y ago
      It does the same thing with a bunch of different words like "committee", "disestablishmentarianism", "dog", "Anaxagoras", and a string I typed by mashing the keyboard, "jwfekduadasjeudapu". It seems fairly general and to perform pretty reliably. (Sometimes the trace is noisier, especially in quants other than the original.) This task is pretty simple and I think can be solved easily with the same kind of statistical pattern matching these models use to write other text.
  - 1y ago
SpicyLemonZest 1y ago
It clearly is an artifact of tokenization, but I don’t think it’s a “just”. The point is precisely that the GPT system architecture cannot reliably close the gap here; it’s almost able to count the number of Bs in a string, there’s no fundamental reason you could not build a correct number-of-Bs mapping for tokens, and indeed it often gets the right answer. But when it doesn’t you can’t always correct it with things like chain of thought reasoning. This matters because it poses a big problem for the (quite large) category of things where people expect LLMs to be useful when they get just a bit better. Why, for example, should I assume that modern LLMs will ever be able to write reliably secure code? Isn’t it plausible that the difference between secure and almost secure runs into some similar problem?
- viraptor 1y ago
  > cannot reliably close the gap here Have you got any proof they're even trying? It's unlikely that's something their real customers are paying for.
  - SpicyLemonZest 1y ago
    I tried to reproduce it again just now, and ChatGPT 5 seems to be a lot more meticulous about running a python script to double-check its work, which it tells me is because it has a warning in its system prompt telling it to. I don't know if that's proof (or even if ChatGPT reliably tells the truth about what's in its system prompt), but given what OpenAI does and doesn't publish it's the closest I could reasonably expect.
- exasperaited 1y ago
  It's like someone has given a bunch of young people hundreds of billions of dollars to build a product that parses HTML documents with regular expressions. It's not in their interest to write off the scheme as provably unworkable at scale, so they keep working on the edge cases until their options vest.
hansvm 1y ago
Common misconception. That just means the algorithm for counting letters can't be as simple as adding 1 for every token. The number of distinct tokens is tiny compared to the parameter space, and it's not infeasible to store a mapping from token type to character count in those weights. If you're fine appealing to less concrete ideas, transformers are arbitrary function approximators, tokenization doesn't change that, and there are proofs of those facts. For any finite-length function (like counting letters in a bounded domain), it's just a matter of having a big enough network and figuring out how to train it correctly. They just haven't bothered.
- viraptor 1y ago
  > They just haven't bothered. Or they don't see the benefit. I'm sure they could train the representation of every token and make spelling perfect. But if you have real users spending money on useful tasks already - how much money would you spend on training answers to meme questions that nobody will pay for. They did it once for the fun headline already and apparently it's not worth repeating.
  - hansvm 1y ago
    That's just a potential explanation for why they haven't bothered. I don't think we're disagreeing.
- zahlman 1y ago
  > The number of distinct tokens is tiny compared to the parameter space, and it's not infeasible to store a mapping from token type to character count in those weights. You seem to suppose that they actually perform addition internally, rather than simply having a model of the concept that humans sometimes do addition and use it to compute results. Why? > For any finite-length function (like counting letters in a bounded domain), it's just a matter of having a big enough network and figuring out how to train it correctly. They just haven't bothered. The problem is that the question space grows exponentially in the length of input. If you want a non-coincidentally-correct answer to "how many t's in 'correct horse battery staple'?" then you need to actually add up the per-token counts.

by amai 1y ago

falcor84 1y ago

Where in the tokenization does the 3rd b come from?

IanCal 1y ago
The tokenisation means they don’t see the letters at all. They see something like this - to convert just some tokens to words How many 538 do you see in 423, 4144, 9890?
- minimaxir 1y ago
  LLMs don’t see token ids, they see token embeddings that map to those ids, and those embeddings are correlated. The hypothetical embeddings of 538, 423, 4144, and 9890 are likely strongly correlated in the process of training the LLM and the downstream LLM should be able to leverage those patterns to solve the question correctly. Even more so since the training process likely has many examples of similar highly correlated embeddings to identify the next similar token.
  - IanCal 1y ago
    But vitally they are not explicitly shown the letters individually and so “count the letters” is a much harder problem to solve than it is for us.

andrewmcwatters 1y ago

No, it's the entire architecture of the model. There's no real reasoning. It seems that reasoning is just a feedback loop on top of existing autocompletion. It's really disingenuous for the industry to call warming tokens for output, "reasoning," as if some autocomplete before more autocomplete is all we needed to solve the issue of consciousness. Edit: Letter frequency apparently has just become another scripted output, like doing arithmetic. LLMs don't have the ability to do this sort of work inherently, so they're trained to offload the task. Edit: This comment appears to be wildly upvoted and downvoted. If you have anything to add besides reactionary voting, please contribute to the discussion.

spwa4 1y ago
I had a fun experience recently. I asked one of my daughters how many r's there are in strawberry. Her answer? Two ... Of course then you ask her to write it and of course things get fixed. But strange.
- andrewmcwatters 1y ago
  I think that's supposed to be the idea of reasoning functionality, but in practice, it just seems to allow responses to continue longer than that would have otherwise by bisecting the output into warming an output and then using maybe what we would consider cached tokens to assist with further contextual lookups. That is to say, you can obtain the same process by talking to "non-reasoning" models.
- furyofantares 1y ago
  To be honest, if a kid asked me how many r's in strawberry, I would assume they were asking how many r's at the end and say 2.
- ziml77 1y ago
  I hate to break it to you but I think your child might actually have gotten swapped in the hospital with an LLM.
exasperaited 1y ago
In ten years time an LLM lawyer will lose a legal case for someone who can no longer afford a real lawyer because there are so few left. And it'll be because the layers of bodges in the model caused it to go crazy, insult the judge and threaten to burn down the courthouse. There will be a series of analytical articles in the mainstream press, the tech industry will write it off as a known problem with tokenisation that they can't fix because nobody really writes code anymore. The LLM megacorp will just add a disclaimer: the software should not be used in legal actions concerning fruit companies and they disclaim all losses.
- Terr_ 1y ago

wongarsu 1y ago

It can spell the word (writing each letter in uppercase followed by a whitespace, which should turn each letter with its whitespace into a separate token). It also has reasoning tokens to use as scratch space, and previous models have demonstrated knowledge of the fact that spelling words is a useful step to counting letters. Tokenization makes the problem difficult, but not solving it is still a reasoning/intelligence issue

pxc 1y ago
Here's an example of what gpt-oss-20b (at the default mxfp4 precision) does with this question: > How many "s"es are in the word "Mississippi"? The "thinking portion" is: > Count letters: M i s s i s s i p p i -> s appears 4 times? Actually Mississippi has s's: positions 3,4,6,7 = 4. The answer is: > The word “Mississippi” contains four letter “s” s. They can indeed do some simple pattern matching on the query, separate the letters out into separate tokens, and count them without having to do something like run code in a sandbox and ask it the answer. The issue here is just that this workaround/strategy is only trained into the "thinking" models, afaict.
- johnecheck 1y ago
  That proves nothing. The fact that Mississippi has 4 "s" is far more likely to be in the training data than the fact that blueberry has 2 "b"s. And now that fact is going to be in the data for the next round of training. We'll need to need to try some other words on the next model.
  - pxc 1y ago
    It does the same thing with a bunch of different words like "committee", "disestablishmentarianism", "dog", "Anaxagoras", and a string I typed by mashing the keyboard, "jwfekduadasjeudapu". It seems fairly general and to perform pretty reliably. (Sometimes the trace is noisier, especially in quants other than the original.) This task is pretty simple and I think can be solved easily with the same kind of statistical pattern matching these models use to write other text.
- 1y ago

SpicyLemonZest 1y ago

It clearly is an artifact of tokenization, but I don’t think it’s a “just”. The point is precisely that the GPT system architecture cannot reliably close the gap here; it’s almost able to count the number of Bs in a string, there’s no fundamental reason you could not build a correct number-of-Bs mapping for tokens, and indeed it often gets the right answer. But when it doesn’t you can’t always correct it with things like chain of thought reasoning. This matters because it poses a big problem for the (quite large) category of things where people expect LLMs to be useful when they get just a bit better. Why, for example, should I assume that modern LLMs will ever be able to write reliably secure code? Isn’t it plausible that the difference between secure and almost secure runs into some similar problem?

viraptor 1y ago
> cannot reliably close the gap here Have you got any proof they're even trying? It's unlikely that's something their real customers are paying for.
- SpicyLemonZest 1y ago
  I tried to reproduce it again just now, and ChatGPT 5 seems to be a lot more meticulous about running a python script to double-check its work, which it tells me is because it has a warning in its system prompt telling it to. I don't know if that's proof (or even if ChatGPT reliably tells the truth about what's in its system prompt), but given what OpenAI does and doesn't publish it's the closest I could reasonably expect.
exasperaited 1y ago
It's like someone has given a bunch of young people hundreds of billions of dollars to build a product that parses HTML documents with regular expressions. It's not in their interest to write off the scheme as provably unworkable at scale, so they keep working on the edge cases until their options vest.

hansvm 1y ago

Common misconception. That just means the algorithm for counting letters can't be as simple as adding 1 for every token. The number of distinct tokens is tiny compared to the parameter space, and it's not infeasible to store a mapping from token type to character count in those weights. If you're fine appealing to less concrete ideas, transformers are arbitrary function approximators, tokenization doesn't change that, and there are proofs of those facts. For any finite-length function (like counting letters in a bounded domain), it's just a matter of having a big enough network and figuring out how to train it correctly. They just haven't bothered.

viraptor 1y ago
> They just haven't bothered. Or they don't see the benefit. I'm sure they could train the representation of every token and make spelling perfect. But if you have real users spending money on useful tasks already - how much money would you spend on training answers to meme questions that nobody will pay for. They did it once for the fun headline already and apparently it's not worth repeating.
- hansvm 1y ago
  That's just a potential explanation for why they haven't bothered. I don't think we're disagreeing.
zahlman 1y ago
> The number of distinct tokens is tiny compared to the parameter space, and it's not infeasible to store a mapping from token type to character count in those weights. You seem to suppose that they actually perform addition internally, rather than simply having a model of the concept that humans sometimes do addition and use it to compute results. Why? > For any finite-length function (like counting letters in a bounded domain), it's just a matter of having a big enough network and figuring out how to train it correctly. They just haven't bothered. The problem is that the question space grows exponentially in the length of input. If you want a non-coincidentally-correct answer to "how many t's in 'correct horse battery staple'?" then you need to actually add up the per-token counts.