3 ms·
GPT-4-Tokenizer-Visualizer
- WilliamBerglund 3y agoHello (author here), I found the OpenAI tool (https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer ) for visualising tokens really useful when working on token optimisation strategies (and other token related tasks). But unfortunately that site doesn't support the tokeniser that GPT4 uses (currently). So I threw together a stackblitz that uses the js-tiktoken to do the same thing. Hopefully it will be useful for someone else too! Try it here: https://react-ts-h9yrjh.stackblitz.io/ https://react-ts-h9yrjh.stackblitz.io/