3 ms·
(Sorry if I ramble I've spent some time on token counting recently and have some opinions on it) It's neat that it's client side only but it's quite feature la
by msp26 3y ago
(Sorry if I ramble I've spent some time on token counting recently and have some opinions on it)
It's neat that it's client side only but it's quite feature lacking compared to https://tiktokenizer.vercel.app/ https://tiktokenizer.vercel.app/ or even a small python script I wrote.
Feedback (Feature parity):
-Identification and separation of individual tokens. Useful for prompting/adjusting logit biases.
-Optional chat formatting
-It's not very responsive to type on, perhaps you could delay the token calc a little
Further suggestions (I don't know how simple you want to keep it):
-Counting tokens for Function Calling json functions
helpful resources:
https://gist.github.com/CGamesPlay/dd4f108f27e2eec145eedf5c717318f5 https://gist.github.com/CGamesPlay/dd4f108f27e2eec145eedf5c7...
https://hmarr.com/blog/counting-openai-tokens/ https://hmarr.com/blog/counting-openai-tokens/