3 ms·
I use my https://github.com/simonw/ttok https://github.com/simonw/ttok command for that - you can pipe stuff into it for a token count. Unfortunately it only u
by simonw 2y ago
I use my https://github.com/simonw/ttok https://github.com/simonw/ttok command for that - you can pipe stuff into it for a token count.
Unfortunately it only uses the OpenAI tokenizers at the moment (via tiktoken), so counts for other models may be inaccurate. I find they tend to be close enough though.