4 ms·
The cost per token served has been falling steadily over the past few years across basically all of the providers. OpenAI dropped the price they charged for o3
by simonw 9mo ago
The cost per token served has been falling steadily over the past few years across basically all of the providers. OpenAI dropped the price they charged for o3 to 1/5th of what it was in June last year thanks to "engineers optimizing inferencing", and plenty of other providers have found cost savings too.
Turns out there was a lot of low-hanging fruit in terms of inference optimization that hadn't been plucked yet.
> A year or more ago, I read that both Anthropic and OpenAI were losing money on every single request even for their paid subscribers
Where did you hear that? It doesn't match my mental model of how this has played out.
- nubg 9mo ago> "engineers optimizing inferencing" are we sure this is not a fancy way of saying quantization?
- embedding-shape 9mo agoOr distilled models, or just slightly smaller models but same architecture. Lots of options, all of them conveniently fitting inside "optimizing inferencing".
- jmalicki 9mo agoA ton of GPU kernels are hugely inefficient. Not saying the numbers are realistic, but look at the 100s of times of gain in the Anthropic performance takehome exam that floated around on here. And if you've worked with pytorch models a lot, having custom fused kernels can be huge. For instance, look at the kind of gains to be had when FlashAttention came out. This isn't just quantization, it's actually just better optimization. Even when it comes to quantization, Blackwell has far better quantization primitives and new floating point types that support row or layer-wise scaling that can quantize with far less quality reduction. There is also a ton of work in the past year on sub-quadratic attention for new models that gets rid of a huge bottleneck, but like quantization can be a tradeoff, and a lot of progress has been made there on moving the Pareto frontier as well. It's almost like when you're spending hundreds of billions on capex for GPUs, you can afford to hire engineers to make them perform better without just nerfing the models with more quantization.
- Der_Einzige 9mo ago"This isn't X, it's Y" with extra steps.
- esafak 9mo agoSomeone made a quality tracker: https://marginlab.ai/trackers/claude-code/ https://marginlab.ai/trackers/claude-code/
- bityard 9mo agoWhen MP3 became popular, people were amazed that you could compress audio to 1/10th its size with minor quality loss. A few decades later, we have audio compression that is much better and higher-quality than MP3, and they took a lot more effort than "MP3 but at a lower bitrate." The same is happening in AI research now.
- oblio 9mo ago> A few decades later, we have audio compression that is much better and higher-quality than MP3 Just curious, which formats and how they compare, storage wise? Also, are you sure it's not just moving the goalposts to CPU usage? Frequently more powerful compression algorithms can't be used because they use lots of processing power, so frequently the biggest gains over 20 years are just... hardware advancements.
- simonw 9mo agoThe o3 optimizations were not quantization, they confirmed this at the time.
- cootsnuck 9mo agoI have not see any reporting or evidence at all that Anthropic or OpenAI is able to make money on inference yet. > Turns out there was a lot of low-hanging fruit in terms of inference optimization that hadn't been plucked yet. That does not mean the frontier labs are pricing their APIs to cover their costs yet. It can both be true that it has gotten cheaper for them to provide inference and that they still are subsidizing inference costs. In fact, I'd argue that's way more likely given that has been precisely the goto strategy for highly-competitive startups for awhile now. Price low to pump adoption and dominate the market, worry about raising prices for financial sustainability later, burn through investor money until then. What no one outside of these frontier labs knows right now is how big the gap is between current pricing and eventual pricing.
- NitpickLawyer 9mo ago> they still are subsidizing inference costs. They are for sure subsidising costs on all you can prompt packages (20-100-200$ /mo). They do that for data gathering mostly, and at a smaller degree for user retention. > evidence at all that Anthropic or OpenAI is able to make money on inference yet. You can infer that from what 3rd party inference providers are charging. The largest open models atm are dsv3 (~650B params) and kimi2.5 (1.2T params). They are being served at 2-2.5-3$ /Mtok. That's sonnet / gpt-mini / gemini3-flash price range. You can make some educates guesses that they get some leeway for model size at the 10-15$/ Mtok prices for their top tier models. So if they are inside some sane model sizes, they are likely making money off of token based APIs.
- sumitkumar 9mo agoIt seems it is true for gemini because they have a humongous sparse model but it isn't so true for the max performance opus-4.5/6 and gpt-5.2/3.
- replwoacause 9mo agoMy experience trying to use Opus 4.5 on the Pro plan has been terrible. It blows up my usage very very fast. I avoid it altogether now. Yes, I know they warn about this, but it's comically fast how quickly it happens.
- topaz0 9mo agoBut a) that's the cost to the user -- we don't know how much loss they're taking on those and b) the number of tokens to serve a similar prompt has been going up, so that the total cost to serve a prompt has been going up in general. Any cost analysis that doesn't mention these is hugely misleading.