11 ms·
Yeah it's BPE. OpenAI has a nice tool that allows you to play with the tokenizer https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer.
by axiom92 4y ago
Yeah it's BPE. OpenAI has a nice tool that allows you to play with the tokenizer https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer.