4 ms·
This is not it or not sufficient explanation. chatGPT is able to read text with letters purposefully scrambled. Like those texts that show how humans have an in
by pa7x1 4y ago
This is not it or not sufficient explanation. chatGPT is able to read text with letters purposefully scrambled. Like those texts that show how humans have an innate ability to read text with letters misplaced. chatGPT can understand the text, figure out what is going on and produce text of the same type. I tried it, precisely to understand if it was working with words as its token.
Interestingly, it cannot produce text with the order or characters reversed or guess that's what is going on. And if you explain it, it's able to produce code that would reverse the characters but is still not able to apply it for long sentences.
- nverno 4y agoDid you try entering one of those scrambled sentences into the tokenizer? It's not tokenizing words exactly- the doc says roughly ~100 tokens => 75 words.
- pa7x1 4y agoI just tried out of curiosity. The jumbled words get broken up into different tokens, the unjumbled words do not get split. https://imgur.com/a/a3zmkIv https://imgur.com/a/a3zmkIv