3 ms·
You are right to be curious. The encoding used by both GPT-3.5 and GPT-4 is called `cl100k_base`, which immediately and correctly suggests that there are about
by lifthrasiir 3y ago
You are right to be curious. The encoding used by both GPT-3.5 and GPT-4 is called `cl100k_base`, which immediately and correctly suggests that there are about 100K tokens.
- cwsx 3y agoAmazing, thanks for the reply, I'm finding some good resources afyer a quick search of `cl100k_base`. If you have any other resources (for anything AI related) please share!
- dchest 3y agoTheir tokenizer is open source: https://github.com/openai/tiktoken https://github.com/openai/tiktoken Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7dd73562b5e211d728723d/tiktoken_ext/openai_public.py https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...
- senseiV 3y agoGPT 2 and 3 used the p50K right? Then GPT-4 used cl100K
- lifthrasiir 3y agoYeah, see [1]. [1] https://github.com/openai/tiktoken/blob/main/tiktoken/model.py https://github.com/openai/tiktoken/blob/main/tiktoken/model....