44 ms·
TikToken doesn't provide a tokenizer that's compatible with LLaMA.
by belladoreai 3y ago
TikToken doesn't provide a tokenizer that's compatible with LLaMA.
- mike_hearn 3y agoAh interesting. What's the difference? Isn't it just finding the minimal mapping of character sequences to numbers?
- belladoreai 3y agoThat's what I thought when I started working on this, but it turns out the answer is no! This approach - "minimal mapping of character sequences to numbers" - can be described as a greedy algorithm. Using this approach will often produce correct results, but not always. I can provide an example: Input string: " grabbed" Tokenize that with the greedy algorithm, you get [17229, 2580] == [" grab", "bed"] Tokenize that with actual LLaMA tokenizer, you get [2646, 1327, 287] == [" gra", "bb", "ed"] Note that the correct tokenizer represents this string with 3 tokens, even though it would be more efficient to represent this string with 2 tokens (yes, those 2 tokens exist in the vocabulary). LLaMA uses SentencePiece Byte-Pair Encoding for tokenization, and it has many weird quirks like this.