2 ms·
Is that because four letters happen to pack into a word on most architectures or is there some other underlying reason? Also Sac and rament together already for
by jacquesm 3y ago
Is that because four letters happen to pack into a word on most architectures or is there some other underlying reason? Also Sac and rament together already form a word so it's kind of logical have the final 'o' as a separate token.
- daveguy 3y agoThe underlying reason is closer to your second observation. Statistically chosen tokens are most likely to be reusable and composable. Much less to do with hardware architecture. Token vocabulary happens before any quantization where processor word size may play a role.
- llm_nerd 3y agoIt's just a function of finding the most efficient solution for encoding a corpus of text into a given vocabulary size. Using something like SentencePiece you can define how big you want your vocabulary to be (the number of discrete tokens), and it will find the best solution of subwords/characters in a sample set. In the case of GPT-3 the vocabulary has a size of 50,257 tokens. GPT-4 increases that past 100k (see https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a09193 https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a0...). It's very similar to compression algos, really. Find recurring sets of characters.