3 ms·
Most LLMs are crippled at tasks regarding character level manipulation because of how the inputs are tokenized (words and numbers are split into common substrin
by yunyu 4y ago
Most LLMs are crippled at tasks regarding character level manipulation because of how the inputs are tokenized (words and numbers are split into common substrings rather than the original characters, 101 -> “10” + “1” vs “1” + “0” + “1”).
If you tried this binary conversion task with a model that takes in raw bytes (for instance, ByT5) it would perform almost perfectly. In GPT’s case, there’s a trade off between having a long context window vs being good at character level tasks, and OpenAI picked the former.
- gdw2 4y agoI tried to use it for help with a wordle. I think the word was EJECT and I had E?E?T. It seemed to understand what I was asking, but came up with 5 and 6 letter suggestions that didn't fit.