4 ms·Tokens consist text both larger and smaller than whole words.by flangola7 3y agoTokens consist text both larger and smaller than whole words.Kerb_ 3y agoThe tokens that make up rare words would still likely be just as rare as the words.flangola7 3y agoAnd yet GPT-4 has no problem crafting new words from whole cloth. If you're making new words rare tokens are useful and the model knows this.