3 ms·
It's open source so you can take a look (I'm not the author): https://github.com/AgentOps-AI/tokencost/blob/main/tokencost/costs.py https://github.com/AgentOps-
by dudeinhawaii 2y ago
It's open source so you can take a look (I'm not the author): https://github.com/AgentOps-AI/tokencost/blob/main/tokencost/costs.py https://github.com/AgentOps-AI/tokencost/blob/main/tokencost...
It looks like tiktoken is the default for most of the methods.
Disclaimer: I didn't fully trace which are being used in each case/model.
- refibrillator 2y ago> # TODO: Add Claude support There are no cases for Claude models yet. I wonder if anyone has run a bunch of messages through Anthropic's API and used the returned token count to approximate the tokenizer?
- GranPC 2y agoSorta: https://github.com/javirandor/anthropic-tokenizer https://github.com/javirandor/anthropic-tokenizer
- simonw 2y agoYeah, I asked here because I dug around in the code and couldn't see how they were doing this, wanted to check I hadn't missed something.