3 ms·
I don't understand how the Claude functionality works. As far as I know Anthropic haven't released the tokenizer for Claude - unlike OpenAI's tiktoken - but yo
by simonw 2y ago
I don't understand how the Claude functionality works.
As far as I know Anthropic haven't released the tokenizer for Claude - unlike OpenAI's tiktoken - but your tool lists the Claude 3 models as supported. How are you counting tokens for those?
- dudeinhawaii 2y agoIt's open source so you can take a look (I'm not the author): https://github.com/AgentOps-AI/tokencost/blob/main/tokencost/costs.py https://github.com/AgentOps-AI/tokencost/blob/main/tokencost... It looks like tiktoken is the default for most of the methods. Disclaimer: I didn't fully trace which are being used in each case/model.
- refibrillator 2y ago> # TODO: Add Claude support There are no cases for Claude models yet. I wonder if anyone has run a bunch of messages through Anthropic's API and used the returned token count to approximate the tokenizer?
- GranPC 2y agoSorta: https://github.com/javirandor/anthropic-tokenizer https://github.com/javirandor/anthropic-tokenizer
- simonw 2y agoYeah, I asked here because I dug around in the code and couldn't see how they were doing this, wanted to check I hadn't missed something.
- Areibman 2y agoAnthropic actually has a Claude 3 tokenizer tucked away in one of their repos: https://github.com/anthropics/anthropic-tokenizer-typescript https://github.com/anthropics/anthropic-tokenizer-typescript At this moment, Tokencost uses the OpenAI tokenizer as a default tokenizer, but this would be a welcome PR!
- simonw 2y ago"This package can be used to count tokens for Anthropic's older models. As of the Claude 3 models, this algorithm is no longer accurate [...]" I've been bugging Anthropic about this for a while, they said that releasing a new tokenizer is not on their current roadmap.
- throwaway211 2y agoImagine a coffee shop refusing to have a price list until after the coffee's been made.
- kevindamm 2y agoIn many countries a taxi won't tell you how much the ride will cost. The first time I traveled to somewhere that negotiated the cost up front it blew my mind. Frequently, contracts will have room for additional charges if circumstances change even a little, or products will have a market rate (fish, equity, etc.). It might seem absurd but variable cost things are not uncommon.
- Breza 2y agoThis is what Washington DC did when I moved here. They theoretically had zones, but in reality it was arbitrary. Moving to meters was an amazing development. Similarly, as LLMs become more and more commonplace, the pricing models will need to be more predictable. My LLM expenses are only around $100/month, but it's a bigger impediment to pushing projects to production when I can't tell the boss exactly how it'll be priced.
- MattGaiser 2y agoIn Oman, every tax fare was basically just of the bills. So 5 rials, 10 rials, or 20 rials. So many different ways people price things around the world.
- _flux 2y agoIn this case there's nothing that's variable, though, and the competition is able to pull it off precisely. Indeed, they themselves were able to do it before!
- J_Shelby_J 2y agoHere you go https://github.com/javirandor/anthropic-tokenizer https://github.com/javirandor/anthropic-tokenizer