7 ms·
Show HN: Token price calculator for 400+ LLMs
Hey HN! Tokencost is a utility library for estimating LLM costs. There are hundreds of different models now, and they all have their own pricing schemes. It’s difficult to keep up with the pricing changes, and it’s even more difficult to estimate how much your prompts and completions will cost until you see the bill.
Tokencost works by counting the number of tokens in prompt and completion messages and multiplying that number by the corresponding model cost. Under the hood, it’s really just a simple cost dictionary and some utility functions for getting the prices right. It also accounts for different tokenizers and float precision errors.
Surprisingly, most model providers don't actually report how much you spend until your bills arrive. We built Tokencost internally at AgentOps to help users track agent spend, and we decided to open source it to help developers avoid nasty bills.
- simonw 2y agoI don't understand how the Claude functionality works. As far as I know Anthropic haven't released the tokenizer for Claude - unlike OpenAI's tiktoken - but your tool lists the Claude 3 models as supported. How are you counting tokens for those?
- dudeinhawaii 2y agoIt's open source so you can take a look (I'm not the author): https://github.com/AgentOps-AI/tokencost/blob/main/tokencost/costs.py https://github.com/AgentOps-AI/tokencost/blob/main/tokencost... It looks like tiktoken is the default for most of the methods. Disclaimer: I didn't fully trace which are being used in each case/model.
- refibrillator 2y ago> # TODO: Add Claude support There are no cases for Claude models yet. I wonder if anyone has run a bunch of messages through Anthropic's API and used the returned token count to approximate the tokenizer?
- GranPC 2y agoSorta: https://github.com/javirandor/anthropic-tokenizer https://github.com/javirandor/anthropic-tokenizer
- simonw 2y agoYeah, I asked here because I dug around in the code and couldn't see how they were doing this, wanted to check I hadn't missed something.
- Areibman 2y agoAnthropic actually has a Claude 3 tokenizer tucked away in one of their repos: https://github.com/anthropics/anthropic-tokenizer-typescript https://github.com/anthropics/anthropic-tokenizer-typescript At this moment, Tokencost uses the OpenAI tokenizer as a default tokenizer, but this would be a welcome PR!
- simonw 2y ago"This package can be used to count tokens for Anthropic's older models. As of the Claude 3 models, this algorithm is no longer accurate [...]" I've been bugging Anthropic about this for a while, they said that releasing a new tokenizer is not on their current roadmap.
- throwaway211 2y agoImagine a coffee shop refusing to have a price list until after the coffee's been made.
- kevindamm 2y agoIn many countries a taxi won't tell you how much the ride will cost. The first time I traveled to somewhere that negotiated the cost up front it blew my mind. Frequently, contracts will have room for additional charges if circumstances change even a little, or products will have a market rate (fish, equity, etc.). It might seem absurd but variable cost things are not uncommon.
- Breza 2y agoThis is what Washington DC did when I moved here. They theoretically had zones, but in reality it was arbitrary. Moving to meters was an amazing development. Similarly, as LLMs become more and more commonplace, the pricing models will need to be more predictable. My LLM expenses are only around $100/month, but it's a bigger impediment to pushing projects to production when I can't tell the boss exactly how it'll be priced.
- 2y ago
- J_Shelby_J 2y agoHere you go https://github.com/javirandor/anthropic-tokenizer https://github.com/javirandor/anthropic-tokenizer
- yelnatz 2y agoCan you do a column and normalize them? Too many zeroes for my blind ass making it hard to compare.
- ryaneager 2y agoYeah a Tokens per $1 column would vastly help the readability.
- qeternity 2y ago$/million tokens is the standard pricing metric.
- Tao3300 2y agostandard ≠ good
- andenacitelli 2y agoYes, but it’s true in the general case. Defaults are usually the defaults for a reason — someone putting thought into what makes sense for most users.
- Tao3300 2y agoNot necessarily so when you're trying to sell stuff...
- ynniv 2y ago"Standard" doesn't imply "good", but that doesn't mean "non-standard" is better. Cost-per-quantity (L/100km) is easier to compare than quantity-per-cost (MPG) because how much you use isn't going to change based on the model. Which is to say, if two local models are both $0.00 per million tokens, they effectively have the same cost. You could argue that you might get better results by throwing out more tokens, but the solution is to add more significant digits to the price per unit.
- ilaksh 2y agoNice. Any plans to add calculations for image input for the models that allow that?
- Areibman 2y agoPerhaps at some point! Right now, we haven't been seeing most demand on the language side of things as multi-modal image really hasn't popped off yet
- pamelafox 2y agoI have a package here that includes calculation for images for OpenAI: https://github.com/pamelafox/openai-messages-token-helper https://github.com/pamelafox/openai-messages-token-helper
- Ilasky 2y agoI dig it! Kind of related, but I made a comparison of LLM API costs vs their leaderboard performance to gauge which models can be more bang for the buck [0] [0] https://llmcompare.net https://llmcompare.net
- SubiculumCode 2y agoSure makes the case for Gemini Pro, doesn't it.
- visarga 2y agoNice, can you make a triangle with (cost, performance, speed)? That would show the tradeoffs.
- Lerc 2y agoWith all the options there seems like an opportunity for a single point API that can take a series of prompts, a budget and a quality hint to distribute batches for most bang for buck. Maybe a small triage AI to decide how effectively models handle certain prompts to preserve spending for the difficult tasks. Does anything like this exist yet?
- curious_cat_163 2y agoI have yet to find a use case where quality can be traded off. Would love to hear what you had in mind.
- Breza 2y agoI've encountered plenty of tasks where lower quality models work quite well. I prefer using Claude 3 Opus, DBRX, or Llama-3, but that level of quality isn't always needed. Here are a few examples. Top story picker. Given a bunch of news stories, pick which one should be the lead story. Data viz color picker. Given a list of categories for a chart, return a color for each one. Windows Start menu. Given a list of installed programs and a query, select the five most likely programs that the user wants.
- mvdtnz 2y agoEvery single use case of LLMs inherently sacrifices quality, whether the developers are willing to admit it or not. I agree with you though that there aren't many use cases where end users would knowingly accept the trade off.
- Lerc 2y agoIt is not so much a drop in quality as there are tasks that every model above a certain threshold will perform equally. Most can do 2+2 = 4. One test prompt I use on LLMs is asking it to produce a JavaScript function that takes an ImageData object and returns a new ImageData object with an all direction Sobel edge detection. Quite a lot of even quite small models can generate functions like this. In general, I don't even think this is a question that needs to be answered. A lot of API providers have different quality/price tiers. The fact that people are using the different tiers should be sufficient to show that at least some people are finding cases where cheaper models are good enough.
- Karrot_Kream 2y agoA whole bunch of the costs are listed as zeroes, with multiple decimal points. I noticed y'all used the Decimal library and tried to hold onto precision so I'm not sure what's going on, but certainly some of the cheaper models just show up as "free".
- jacobglowbom 2y agoNice. Does it add Vision costs too?
- pamelafox 2y agoI have a package here that includes calculation for images for OpenAI: https://github.com/pamelafox/openai-messages-token-helper https://github.com/pamelafox/openai-messages-token-helper
- zackfield 2y agoVery cool! Is this cost directory you're using the best source for historical cost per 1M tokens? https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json https://github.com/BerriAI/litellm/blob/main/model_prices_an...
- Areibman 2y agoBest one I've found out there. If there's another, very open to replacing
- refulgentis 2y ago[flagged]
- visarga 2y agoYou don't like a repo, you don't use it. Stop shaming people for their open source repos.
- oopsallmagic 2y agoWhen did the software community get so bad at handling legitimate critique?
- fbdab103 2y ago"You should really be ashamed of yourself for doing this" is an inappropriate response for basically anything but kicking puppies.
- anoncareer0212 2y agoThat sounds somewhat specious, lying about what you support in your cost calculation library is a pretty big oof. It's hard to rank it versus kicking puppys, but, I don't think we have to stack-rank bad things to figure out if it's okay to call out unethical behavior.
- Areibman 2y agoThis is unnecessarily harsh. Not every model has a publicly available tokenizer, and using a fallback like cl100k is usually a decent enough estimator from my experience. Besides, there's a warning message for when you specify a model without a known tokenizer. If you're upset with the implementation, you can always raise an issue or fix it yourself
- refulgentis 2y ago> This is unnecessarily harsh. Which part? All I can tease out from your comment are "the lies are impossible" (agreed!) and "close enough afaik". (it's not, the closest in the Big 5 has percent error of 32%, see end of comment. ex. GPT4o has a tokenizer with 2x the vocab so you'd expect ~1/2 the tokens) > Not every model has a publicly available tokenizer, Right. Ex. Claude 3s and Geminis. So why are Claude 3s and Geminis listed as supported models? > using a fallback CL100K isn't a fallback, its the only tokenizer. > like cl100k is usually a decent enough estimator from my experience. I'm very surprised to hear this, per stats demonstrating minimum error of 32%. > If you're upset with the implementation, you can always raise an issue I'm not "upset with the implementation", I'm sharing that the claims about being able to make financial calculations for 400 different LLMs is lying. > or fix it yourself How? As you pointed out, its unfixable for at least some subset of the ones they're claiming, ex. Gemini and Claude 3s. Let's pretend it was possible. Why? If someone puts out a library making wildly false claims, is th right thing to do to stay quiet and fix the library making false claims until its claims are true? > usually a decent enough estimator No, not for financial things certainly, which is the stated core purpose of the library. As promised, data: I picked the simplest example from my unit tests because you won't believe the divergence on larger ones. OpenAI (CL100K) - 18 in/1 out = 19. Gemini 1.5 - 41 in/14 out = 55. (65% error) Claude 3 - 21 in/4 out = 25. (24% error) Llama 3 - 23 in/5 out = 28. (32% error) Mistral - 10 in/3 out = 13. (46% error)
- oopsallmagic 2y agoCan we get conversions for kg of CO2 emitted, too?
- _flux 2y agoIt would be nice, but how do we get this information required for the conversion?
- jaredliu233 2y agowow, this is really useful!! Just the price list alone has given me a lot of inspiration, thank you
- yumaueno 2y agoWhat a nice product! I think the way to count tokens depends on the language, but is this only supported in English?
- lgessler 2y agoMost LLMs determine their token inventories by using byte-pair encoding, which algorithmically induces sub-word tokens from a body of text. So even in English you might see a word like "proselytization" tokenized apart into "_pro", "selyt", "iz", "ation", and non-English languages will probably (depending on their proportional representation in the training corpus) also receive token allocations in the BPE vocabulary. Here's actual output from the GPT-4o tokenizer for English and Hindi: >>> [enc.decode([x]) for x in enc.encode("proselytization")] ['pros', 'ely', 't', 'ization'] >>> [enc.decode([x]) for x in enc.encode("पर्यावरणवाद")] ['पर', '्य', 'ावरण', 'वाद']
- sakex 2y agoAn interesting parameter that I don't read about a lot is vocab size. A larger vocab means you will need to generate less tokens for the same word on average, also the context window will be larger. This means that a model with a large vocab might be more expensive on a per token basis, but would generate less tokens for the same sentence, making it cheaper overall. This should be taken into consideration when comparing API prices.
- weird-eye-issue 2y agoYeah... it obviously uses the appropriate tokenizer
- spencerchubb 2y agoSome companies like Anthropic haven't publicly released the tokenizer used in their API, thereby making it impossible for this library to use the appropriate tokenizer in all cases. Be careful about how you use the word 'obviously'
- deleted 2y ago[deleted]
- neverokay 2y agoGemini also only charges for output tokens, not sure if that’s considered. All in all this is something I was looking for or was roughly going to do to compare costs. Cool stuff.
- pamelafox 2y agoAre you also accounting for costs of sending images and function calls? I didn't see that when I looked through the code. I developed this package so that I could count those sorts of calls as well: https://github.com/pamelafox/openai-messages-token-helper https://github.com/pamelafox/openai-messages-token-helper
- J_Shelby_J 2y agoWould anybody be interested in this for Rust? I already do everything this library does with the exception of returning the price in my LLM utils crate [1]. I do this just to count tokens to ensure prompts stay within limits. And I also support non-open ai tokenizers. So adding a price calculator function would be trivial. [1] https://github.com/ShelbyJenkins/llm_utils https://github.com/ShelbyJenkins/llm_utils
- armen99 2y agoThis is great project! I would love to see something that calculates training costs as well.