4 ms·

Please anyone correct me if I'm wrong: LLMs cannot solve this kind of riddle. This has nothing to do with their capabilities for logical reasoning, but with the

by planb 3y ago

Please anyone correct me if I'm wrong: LLMs cannot solve this kind of riddle. This has nothing to do with their capabilities for logical reasoning, but with the way words are represented as tokens. While they might know that "apples" has two syllables because that is mentioned somewhere in their training data, if you make up a fruit "bratush" a human will see that as two syllables, but this might be 1 to 7 tokens to a LLM without any information about the word itself.

mborch 3y ago
ChatGPT knows how to answer this question, but how I don't know. Perhaps it's programmed to answer that question?
huytersd 3y ago
Well I tried it out in GPT4 with made up words- Tabitha likes bratush but not zot. She likes protel but not kig, and she likes motsic but not pez. Following the same rule, will she like tridos or kip Given the examples, one speculative pattern could be that Tabitha likes words with at least two syllables or a certain complexity in structure. Therefore, following this speculative rule, Tabitha might like “tridos” more than “kip.”
- bitmovements 3y ago
  Zot is a word already being used in the world: verb. (slang) To zap, kill, or destroy. So is protel: https://en.m.wikipedia.org/wiki/Protel https://en.m.wikipedia.org/wiki/Protel So is kig: https://en.m.wiktionary.org/wiki/kig https://en.m.wiktionary.org/wiki/kig Pez is a well known brand name in America. Kip is commonly used name in America. Motsic is a fairly common last name from searching. Tridos is used all over the internet as a brand name so this all seems probable to be in the training data. These words are not new nor are they made up.
ComputerGuru 3y ago
GPT-4 has a really good tokenizer that is able to retain and use more information about input tokens than one might naively think.
- planb 3y ago
  Interesting. Given the example with made up words someone posted here it really looks like I was wrong.
andrewla 3y ago
The amazing thing about emergent behavior in LLMs is that they are able to answer questions like these. I don't think it is completely understood how exactly they do this, but there's little doubt that they do.
- boppo1 3y ago
  Do you have any sources that prove this is true?
  - jerbear4328 3y ago
    This looks pretty good to me: https://chat.openai.com/share/040ac123-c690-4274-8216-6ae09167ef13 https://chat.openai.com/share/040ac123-c690-4274-8216-6ae091...
    - bitmovements 3y ago
      Your word is not made up, nor are some of the others in the sample like”pez.” I don’t think this test proves what you think it does. Bratush come up quite a bit in the internet: https://www.thefreedictionary.com/words-that-start-with-bratush https://www.thefreedictionary.com/words-that-start-with-brat...
viraptor 3y ago
LLM can solve this for all tokens where it got to learn how many syllables are in that token or a combination. If you trained it to work on single letters only it would do better at that task than word chunks (same for math and single digits). It will generalise to new words if the token level knowledge is there. Whether this means it can or cannot solve that kind of riddle is up for your interpretation. I understand square root and can calculate square root of 16, but not of 738284.7280594873. (in a reasonable, bounded time) Can I solve square roots?

by planb 3y ago

mborch 3y ago

ChatGPT knows how to answer this question, but how I don't know. Perhaps it's programmed to answer that question?

huytersd 3y ago

Well I tried it out in GPT4 with made up words- Tabitha likes bratush but not zot. She likes protel but not kig, and she likes motsic but not pez. Following the same rule, will she like tridos or kip Given the examples, one speculative pattern could be that Tabitha likes words with at least two syllables or a certain complexity in structure. Therefore, following this speculative rule, Tabitha might like “tridos” more than “kip.”

bitmovements 3y ago
Zot is a word already being used in the world: verb. (slang) To zap, kill, or destroy. So is protel: https://en.m.wikipedia.org/wiki/Protel https://en.m.wikipedia.org/wiki/Protel So is kig: https://en.m.wiktionary.org/wiki/kig https://en.m.wiktionary.org/wiki/kig Pez is a well known brand name in America. Kip is commonly used name in America. Motsic is a fairly common last name from searching. Tridos is used all over the internet as a brand name so this all seems probable to be in the training data. These words are not new nor are they made up.

ComputerGuru 3y ago

GPT-4 has a really good tokenizer that is able to retain and use more information about input tokens than one might naively think.

planb 3y ago
Interesting. Given the example with made up words someone posted here it really looks like I was wrong.

andrewla 3y ago

The amazing thing about emergent behavior in LLMs is that they are able to answer questions like these. I don't think it is completely understood how exactly they do this, but there's little doubt that they do.

boppo1 3y ago
Do you have any sources that prove this is true?
- jerbear4328 3y ago
  This looks pretty good to me: https://chat.openai.com/share/040ac123-c690-4274-8216-6ae09167ef13 https://chat.openai.com/share/040ac123-c690-4274-8216-6ae091...
  - bitmovements 3y ago
    Your word is not made up, nor are some of the others in the sample like”pez.” I don’t think this test proves what you think it does. Bratush come up quite a bit in the internet: https://www.thefreedictionary.com/words-that-start-with-bratush https://www.thefreedictionary.com/words-that-start-with-brat...

viraptor 3y ago

LLM can solve this for all tokens where it got to learn how many syllables are in that token or a combination. If you trained it to work on single letters only it would do better at that task than word chunks (same for math and single digits). It will generalise to new words if the token level knowledge is there. Whether this means it can or cannot solve that kind of riddle is up for your interpretation. I understand square root and can calculate square root of 16, but not of 738284.7280594873. (in a reasonable, bounded time) Can I solve square roots?