2 ms·
You can paste text and immediately see how it's split into tokens, along with token IDs, counts, and other details. It's useful for understanding why prompts co
by laurentiurad 3mo ago
You can paste text and immediately see how it's split into tokens, along with token IDs, counts, and other details. It's useful for understanding why prompts cost what they do, why context limits behave the way they do, and why models sometimes split words in unexpected places.
I made it because I kept explaining tokenization to friends and realized there wasn't a simple, interactive tool that focused on learning rather than documentation.
I'd love feedback on:
Features that would make it more useful
UI improvements
Bugs or edge cases you've found
Hope it's useful!
- hk__2 3mo ago> I made it *I asked a LLM to make it
- JojiDoesCode 3mo agoHonestly this person's whole github pages screams tech grifter
- deleted 3mo ago[deleted]
- laurentiurad 3mo agoI am not even asking for donations, what are you talking about?