4 ms·

Their tokenizer is open source: https://github.com/openai/tiktoken https://github.com/openai/tiktoken Data files that contain vocabulary are listed here: https

by dchest 3y ago

Their tokenizer is open source: https://github.com/openai/tiktoken https://github.com/openai/tiktoken Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7dd73562b5e211d728723d/tiktoken_ext/openai_public.py https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...