4 ms·
Made a GPT-3.5-Turbo and GPT-4 Tokenizer
- toshvelaga 3y agoFor the longest time I haven't been able to find a tokenizer online similar to the one for GPT-3 for other models. So I made one and put it up online for free for anyone. It's built on top of the tiktoken library and is basically just a lambda function in the backend. You could just download the library directly but I hope this saves someone time if they need the token ids for logit bias.