3 ms·
One interesting fact I stumbled upon recently is that GPT2Tokenizer library and Tiktoken library produces the same number of tokens for `text-davinci-003` model
by nextworddev 4y ago
One interesting fact I stumbled upon recently is that GPT2Tokenizer library and Tiktoken library produces the same number of tokens for `text-davinci-003` model, despite GPT2Tokenizer being GPT2 and text-davinci-003 being GPT3.5.
For code, however, Tiktoken library and GPT2Tokenizer produce different tokenizations.
- shevis 4y agoKey difference here is in tokenization encoders. The newer models make use of the `cl100k_base` encoding.