4 ms·
Some models are heavily subsidized. Total params & active params are better measurement of inference cost.
by xiaoyu2006 4mo ago
Some models are heavily subsidized. Total params & active params are better measurement of inference cost.
- simianwords 4mo agoNo models are subsidised -- there are lots of third party hosting services that will still run at breakeven/profit. (except Deepseek after discount)
- stymaar 4mo ago> No models are subsidised We have no proof in either direction, it's not like we had access to their financial numbers in details. And the pricing itself muddies the water, as input tokens that are already in the KV cache are practically free for the provider, whereas other tokens are expensive. So they could still make money overall thanks to people having multi-turn conversation (and as such, paying multiple times for the same token), but lose money on actual compute done. > there are lots of third party hosting services that will still run at breakeven/profit. How can you be sure that they are making profit directly from token price, and are not billing at marginal cost (i.e. electricity price, without counting the cost of the GPUs) and aiming to make a profit later on from the valuable training data that they are collecting in the process?
- simianwords 4mo ago> How can you be sure You are free to believe that they are doing all this. Or you can simply believe the intuition that models are getting cheaper by the day. I can run Gemma 4 31B from my laptop today.
- stymaar 4mo ago> Or you can simply believe the intuition Sure, you can believe you intuition as much as you want, but telling strangers over the internet that they are wrong because “I trust my intuition” is… awkward.
- simianwords 4mo agoAt some point it does come to intuition. Even if the companies IPO and share their financials, you can always argue that they might be lying.
- stymaar 4mo agoAgain, there's a difference between relying on intuition in your life (which we all do, lacking perfect information that would allow us to avoid relying on it), and telling someone they are wrong because your intuition says so.
- jgalt212 4mo ago> as input tokens that are already in the KV cache are practically free for the provider, not at today's RAM prices.
- stymaar 4mo agoRAM price don't change anything. You can't fit an infinity of tokens in the KV cache, but the ones that are in there when a user request them are still practically free.