3 ms·
Amazing, thanks for the reply, I'm finding some good resources afyer a quick search of `cl100k_base`. If you have any other resources (for anything AI related)
by cwsx 3y ago
Amazing, thanks for the reply, I'm finding some good resources afyer a quick search of `cl100k_base`.
If you have any other resources (for anything AI related) please share!
- dchest 3y agoTheir tokenizer is open source: https://github.com/openai/tiktoken https://github.com/openai/tiktoken Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7dd73562b5e211d728723d/tiktoken_ext/openai_public.py https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...