3 ms·
One of the reasons is because to ChatGPT, each word is a token. It only knows how many letters are in a word because it has been trained to know that about each
by cultureswitch 3y ago
One of the reasons is because to ChatGPT, each word is a token. It only knows how many letters are in a word because it has been trained to know that about each word individually.
- rvnx 3y agoYou can see it by yourself here if you are interested: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer The tokens are usually not matching the length of a word