3 ms·
It's just a function of finding the most efficient solution for encoding a corpus of text into a given vocabulary size. Using something like SentencePiece you c
by llm_nerd 3y ago
It's just a function of finding the most efficient solution for encoding a corpus of text into a given vocabulary size. Using something like SentencePiece you can define how big you want your vocabulary to be (the number of discrete tokens), and it will find the best solution of subwords/characters in a sample set.
In the case of GPT-3 the vocabulary has a size of 50,257 tokens. GPT-4 increases that past 100k (see https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a09193 https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a0...).
It's very similar to compression algos, really. Find recurring sets of characters.