3 ms·
Count tokens used by GPT-4 and Llama for large texts (> 50k characters)
- xenova 3y agoThis web-app fixes the two main problems of OpenAI's tokenizer playground: (1) being capped at 50k characters, and (2) not supporting GPT-4/GPT-3.5 tokenizers. Everything runs in-browser thanks to Transformers.js.