3 ms·
Have a look at https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer which uses javascript reimplementation of the GPT-2 / GPT-3 BPE tokenizer. In
by mnks 4y ago
Have a look at https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer which uses javascript reimplementation of the GPT-2 / GPT-3 BPE tokenizer. In this case it's [31373, 995].