3 ms·

We noticed that this webpage is out of date for recent models so we (diagram.com) commissioned a better one that lets you pick any of OpenAI's models including

We noticed that this webpage is out of date for recent models so we (diagram.com) commissioned a better one that lets you pick any of OpenAI's models including chats: https://tiktokenizer.vercel.app/ https://tiktokenizer.vercel.app/

JonathanFly 4y ago
Wow, I can't thank you enough for this. Somehow I never noticed that GPT-4 doesn't use separate tokens for each tab like 3.5. I was wasting a lot of time minimizing excessively tabbed code to save on the token count! Like seriously way too much time, all based on a bad assumption. https://twitter.com/jonathanfly/status/1643633463260577794 https://twitter.com/jonathanfly/status/1643633463260577794