3 ms·
What are people using to check token length of code bases? I'd like to point certain app folders to a local LLM, but no idea how that stuff is calculated? Seems
by forgingahead 2y ago
What are people using to check token length of code bases? I'd like to point certain app folders to a local LLM, but no idea how that stuff is calculated? Seems like some strategic prompting (eg: this is a rails app, here is the folder structure with file names, and btw here are the actual files to parse) would be more efficient than just giving it the full app folder? No point giving it stuff from /lib and /vendor for the most part I reckon.
- simonw 2y agoI use my https://github.com/simonw/ttok https://github.com/simonw/ttok command for that - you can pipe stuff into it for a token count. Unfortunately it only uses the OpenAI tokenizers at the moment (via tiktoken), so counts for other models may be inaccurate. I find they tend to be close enough though.
- sumedh 2y agoYou can try Gemini Token count. https://ai.google.dev/api/tokens https://ai.google.dev/api/tokens
- xyc 2y agoYou can use llama.cpp server's tokenize endpoint to tokenize and count the tokens: https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md#post-tokenize-tokenize-a-given-text https://github.com/ggerganov/llama.cpp/blob/master/examples/...