5 ms·

It's a fair criticism, and ChatGPT does better, but this isn't a great test of model quality. All LLMS that rely on tokenization struggle with being introspecti

It's a fair criticism, and ChatGPT does better, but this isn't a great test of model quality. All LLMS that rely on tokenization struggle with being introspective on language. Try asking chatGPT to count how many e's are in a sentence, or to list all words that start with "to" and end wide "de". I haven't heard anyone describe the phenomenon clearly, but I expect it is a challenge with reasoning over both intent of the prompt and specific token IDs.

mckirk 4y ago
The problem is that these models do not have any working memory they could use to carry out such tasks, which are on a meta-level when seen from a language perspective. They can only go with their 'gut instinct' for selecting the next word, they can't 'consider and ponder the problem internally' first.
- jameslevy 4y ago
  Perhaps prompting the model to first describe its approach to answering the question. This type of chain-of-thought technique can yield better results.
- sp332 4y ago
  The problem is that the input is tokenized before the model gets it as input. It does not see the individual letters "t" + "o". It gets one single token, #1462. The word "toe" is another single token, #44579. Maybe over time it could learn from context that inputs that start with #44579 also satisfy the constraint of starting with #1462, but that's a lot of work and it's not going to happen for all combinations of letters.
chaxor 4y ago
It's not a character based model (likely - although it's closed source so anything is technically possible behind the scenes) so this makes some sense. The system can infer some relationships, which may be why 'agy' is conflated with 'agi' interestingly, but the tokenization process yields sequences of 'symbols' or indexes that are decided to English - so the system has a more difficult task when asked about 'e's (probably something like token 4893) and has to determine which tokens (e.g. [358,284840, 58292, 4830104, 57282, 4829193, 58282, 384, 24945] contain 'e's or token 4893). None of them do directly it seems - but 58292 may be 'ee' - so you would get this wrong as well.
ksaj 4y ago
You can't ask ChatGPT to count something and expect that it can answer correctly, because it does not have counting logic. It is a language model, not a math model. People use this to "prove" hallucinations, but when you ask it something that is within it's programmed abilities, you get something at least close to what you want. Having said that, here are the words ChatGPT gave me for the same prompt: Magi Nagi Sagi Yagi Adagi Galagi Tegagi Sigikagi Tagi Wagagi It missed Unagi, surprisingly. But it is still leagues ahead of the response primordialsoup got from Lamini.
- brookst 4y ago
  It's true that ChatGPT is not designed for counting and struggles with it in general. But my point was that ChatGPT, like any tokenized LLM, doesn't even have the concept of letters. The prompt "how many e's in this sentence" is rendered as the tokens [4919, 867, 304, 338, 287, 428, 6827]. There just isn't a pathway for it to consider the letters that make up those tokens. I'm a little surprised it did that well on your prompt, which is rendered as [10919, 2456, 886, 287, 556, 72]. The interesting thing here is that 556 = " ag" (with leading space) and 72 = "i". So I'm not sure how got to those words. "Wagagi" is tokens [54, 363, 18013], so somehow it is seeing that token 18013 is what you get when you combine 556 and 72? That seems really weird. I'd love clarification from someone deeper into LLMs and tokenization.
  - ncraig 4y ago
    This is an excellent question. I wonder if it's something like [1] on letter composition rather than meaning. [1] https://arxiv.org/pdf/1810.04882.pdf https://arxiv.org/pdf/1810.04882.pdf
  - DougBTX 4y ago
    In a prompt, can you just tell the model which letters make up each token? Eg a list of ag = a g etc. I imagine a dictionary of that for all tokens in the training data would help.
    - deleted 4y ago
      [deleted]
two_in_one 4y ago
well, here we go, ChatGPT in GPT-4 mode: There are 7 instances of the letter 'e' in the sentence: "Try asking chatGPT to count how many e's are in a sentence." another one: The words with the letter 'e' from the sentence "Try asking chatGPT to count how many e's are in a sentence" are: asking sentence and another, notice the last one: Here are some English words containing three instances of the letter 'e': Nevertheless Extreme Relevance Precedence Residence Easement Demeanor Please note that this is not an exhaustive list, but these examples should give you an idea of words with three 'e's in them. What surprising here is that it's still capable of writing hundred lines of python code..
- cassianoleal 4y ago
  On the last list, the only word that does not comply with the constraints (having 3 'e's) is "Demeanor", which has only 2. Not great but also not as horrible as you make it sound.
  - spdustin 4y ago
    Nevertheless, it got 2 out of 7 wrong, not just one.
    - cassianoleal 4y ago
      "Nevertheless" does have 3 'e's. In fact, it even has one more than 3. So does "precedence". It was never specified "exactly 3".
kevviiinn 4y ago
GPT-4 did just fine when I asked it to name words that end in agi so I don't think your argument holds

mckirk 4y ago

The problem is that these models do not have any working memory they could use to carry out such tasks, which are on a meta-level when seen from a language perspective. They can only go with their 'gut instinct' for selecting the next word, they can't 'consider and ponder the problem internally' first.

jameslevy 4y ago
Perhaps prompting the model to first describe its approach to answering the question. This type of chain-of-thought technique can yield better results.
sp332 4y ago
The problem is that the input is tokenized before the model gets it as input. It does not see the individual letters "t" + "o". It gets one single token, #1462. The word "toe" is another single token, #44579. Maybe over time it could learn from context that inputs that start with #44579 also satisfy the constraint of starting with #1462, but that's a lot of work and it's not going to happen for all combinations of letters.

chaxor 4y ago

It's not a character based model (likely - although it's closed source so anything is technically possible behind the scenes) so this makes some sense. The system can infer some relationships, which may be why 'agy' is conflated with 'agi' interestingly, but the tokenization process yields sequences of 'symbols' or indexes that are decided to English - so the system has a more difficult task when asked about 'e's (probably something like token 4893) and has to determine which tokens (e.g. [358,284840, 58292, 4830104, 57282, 4829193, 58282, 384, 24945] contain 'e's or token 4893). None of them do directly it seems - but 58292 may be 'ee' - so you would get this wrong as well.

ksaj 4y ago

You can't ask ChatGPT to count something and expect that it can answer correctly, because it does not have counting logic. It is a language model, not a math model. People use this to "prove" hallucinations, but when you ask it something that is within it's programmed abilities, you get something at least close to what you want. Having said that, here are the words ChatGPT gave me for the same prompt: Magi Nagi Sagi Yagi Adagi Galagi Tegagi Sigikagi Tagi Wagagi It missed Unagi, surprisingly. But it is still leagues ahead of the response primordialsoup got from Lamini.

brookst 4y ago
It's true that ChatGPT is not designed for counting and struggles with it in general. But my point was that ChatGPT, like any tokenized LLM, doesn't even have the concept of letters. The prompt "how many e's in this sentence" is rendered as the tokens [4919, 867, 304, 338, 287, 428, 6827]. There just isn't a pathway for it to consider the letters that make up those tokens. I'm a little surprised it did that well on your prompt, which is rendered as [10919, 2456, 886, 287, 556, 72]. The interesting thing here is that 556 = " ag" (with leading space) and 72 = "i". So I'm not sure how got to those words. "Wagagi" is tokens [54, 363, 18013], so somehow it is seeing that token 18013 is what you get when you combine 556 and 72? That seems really weird. I'd love clarification from someone deeper into LLMs and tokenization.
- ncraig 4y ago
  This is an excellent question. I wonder if it's something like [1] on letter composition rather than meaning. [1] https://arxiv.org/pdf/1810.04882.pdf https://arxiv.org/pdf/1810.04882.pdf
- DougBTX 4y ago
  In a prompt, can you just tell the model which letters make up each token? Eg a list of ag = a g etc. I imagine a dictionary of that for all tokens in the training data would help.
  - deleted 4y ago
    [deleted]

two_in_one 4y ago

well, here we go, ChatGPT in GPT-4 mode: There are 7 instances of the letter 'e' in the sentence: "Try asking chatGPT to count how many e's are in a sentence." another one: The words with the letter 'e' from the sentence "Try asking chatGPT to count how many e's are in a sentence" are: asking sentence and another, notice the last one: Here are some English words containing three instances of the letter 'e': Nevertheless Extreme Relevance Precedence Residence Easement Demeanor Please note that this is not an exhaustive list, but these examples should give you an idea of words with three 'e's in them. What surprising here is that it's still capable of writing hundred lines of python code..

cassianoleal 4y ago
On the last list, the only word that does not comply with the constraints (having 3 'e's) is "Demeanor", which has only 2. Not great but also not as horrible as you make it sound.
- spdustin 4y ago
  Nevertheless, it got 2 out of 7 wrong, not just one.
  - cassianoleal 4y ago
    "Nevertheless" does have 3 'e's. In fact, it even has one more than 3. So does "precedence". It was never specified "exactly 3".

kevviiinn 4y ago

GPT-4 did just fine when I asked it to name words that end in agi so I don't think your argument holds