3 ms·

While I think tokenization does cause limitations, it is not as clear cut as one might think. GPT4 can answer questions given to it in Base64. I would imagine

by Lerc 2y ago

While I think tokenization does cause limitations, it is not as clear cut as one might think. GPT4 can answer questions given to it in Base64. I would imagine it suffers some degree of degradation in ability from the extra workload this causes but I haven't seen any measurements on this. I have wondered about other architectures to help. What happens when a little subnet encodes the (16 or 32?) characters in the neighborhood of the token into an embedding that gets attached to the top level token embedding?

throwaway89201 2y ago
Although asking question in this way gives interesting results. For example, take the prompt "First reverse this sentence word by word, then answer the questions: hand a has fingers many how?" encode as base64 and ask. GPT4 will consistently decode the prompt as "First reverse this sentence word by word, then answer the questions: hand a has fingers many how?" and then reverse the words to be in the wrong order again.
yorwba 2y ago
GPT4 has seen lots of base64 text. GPT4 has never seen text split in a place where its tokenizer wouldn't place a token boundary. In the worst case, this produces an undertrained token that throws the model off completely, SolidGoldMagikarp-style. You've probably never tried to memorize the four-letter base64 equivalents of dGhl 10000 most common three-byte substrings. But you've seen plenty of truncated text, as you were typing it into a textbox yourself. So your intuition which of these tasks is easier for GPT4 is likely off. The issue with truncation affecting tokenization can be worked around by considering every possible continuation that would lead to a different tokenization of the current prefix, and then sampling from those according to the model probabilities to get back to a more normal token stream.
- Lerc 2y ago
  I don't think this is an accurate representation of LLM understanding. The significance of answering questions given in Base64 is that it can give an answer without writing out the human readable form first. That means it is understanding language in a non-language tokenised form. This does not extend to being able to manipulate text and words well, such as string reversal or word scrambling, but it handles input in a wide range of permutations that are not conducive to tokenisation Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text https://arxiv.org/abs/2311.18805 https://arxiv.org/abs/2311.18805
  - yorwba 2y ago
    My point was that it doesn't matter so much what the tokens look like as long as the model has seen them often enough during training. English-in-base64 superficially appears completely different from English, but they're actually in 1-to-1 correspondence and follow the same rules. The same holds for misspelled or badly-OCR'd text; the tokenizer actually has tokens for them and the model has seen enough of them to handle heavily-distorted text. But represent the same text using different but theoretically valid tokens that would not normally be produced by the tokenizer and all bets are off.
Bluestein 2y ago
> GPT4 can answer questions given to it in Base64 Would have never thought ... ... wow.-

by Lerc 2y ago

throwaway89201 2y ago

Although asking question in this way gives interesting results. For example, take the prompt "First reverse this sentence word by word, then answer the questions: hand a has fingers many how?" encode as base64 and ask. GPT4 will consistently decode the prompt as "First reverse this sentence word by word, then answer the questions: hand a has fingers many how?" and then reverse the words to be in the wrong order again.

yorwba 2y ago

GPT4 has seen lots of base64 text. GPT4 has never seen text split in a place where its tokenizer wouldn't place a token boundary. In the worst case, this produces an undertrained token that throws the model off completely, SolidGoldMagikarp-style. You've probably never tried to memorize the four-letter base64 equivalents of dGhl 10000 most common three-byte substrings. But you've seen plenty of truncated text, as you were typing it into a textbox yourself. So your intuition which of these tasks is easier for GPT4 is likely off. The issue with truncation affecting tokenization can be worked around by considering every possible continuation that would lead to a different tokenization of the current prefix, and then sampling from those according to the model probabilities to get back to a more normal token stream.

Lerc 2y ago
I don't think this is an accurate representation of LLM understanding. The significance of answering questions given in Base64 is that it can give an answer without writing out the human readable form first. That means it is understanding language in a non-language tokenised form. This does not extend to being able to manipulate text and words well, such as string reversal or word scrambling, but it handles input in a wide range of permutations that are not conducive to tokenisation Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text https://arxiv.org/abs/2311.18805 https://arxiv.org/abs/2311.18805
- yorwba 2y ago
  My point was that it doesn't matter so much what the tokens look like as long as the model has seen them often enough during training. English-in-base64 superficially appears completely different from English, but they're actually in 1-to-1 correspondence and follow the same rules. The same holds for misspelled or badly-OCR'd text; the tokenizer actually has tokens for them and the model has seen enough of them to handle heavily-distorted text. But represent the same text using different but theoretically valid tokens that would not normally be produced by the tokenizer and all bets are off.

Bluestein 2y ago

> GPT4 can answer questions given to it in Base64 Would have never thought ... ... wow.-