If an example would be helpful, OpenAI's tokenizer is publicly usable on their website: https://platform.openai.com/tokenizer https://platform.openai.com/token

If an example would be helpful, OpenAI's tokenizer is publicly usable on their website: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer You can drop sample text in there and visually see how it is split into tokens. The GPT2/3 tokenizer uses about 50k unique tokens that were learned to be an efficient representation of the training data.