8 ms·
The llama 3.0, 3.1, & 3.2 all use the TikToken tokenizer which is the open source openai tokenizer.
by foxhop 2y ago
The llama 3.0, 3.1, & 3.2 all use the TikToken tokenizer which is the open source openai tokenizer.
- littlestymaar 2y agoGP is talking about context windows, not the number of token used by the tokenizer.
- sva_ 2y agoSomewhat confusingly, it appears the tokenizer vocabulary as well as the context length are both 128k tokens!
- littlestymaar 2y agoYup, that's why I wanted to clarify things.