6 ms·
This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T
by NitpickLawyer 3mo ago
This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some range of $/MTok for a 3T model. Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing".
Also interesting to see what effort it will take to fine-tune this beast. The latest AISI benchmarks on cybersec place it above glm5.2, but still way way behind SotA closed models. Some fine-tuning might be needed here. Also, interesting to see if Cursor does another training round on it, to directly compare it w/ kimi2.6/2.7 fine-tunes (composer series) and grok4.5.
Also also, interesting to see if someone takes on distilling (proper distillation, w/ training the entire distribution) from this into smaller models. (dsv4-kimi should be really good, since dsv4 is very cheap to serve)
- dist-epoch 3mo ago> if "labs are subsidising tokens on API pricing" > SemiAnalysis estimates that Anthropic's current blended gross margin has risen to the mid-60% range, with the API business gross margin exceeding 80% Of course, people will insist "they are lying", "why should we believe them, it's well known they subsidize API pricing", ... https://newsletter.semianalysis.com/p/anthropic-3q26-profit-over-1b-the https://newsletter.semianalysis.com/p/anthropic-3q26-profit-... https://finance.biggo.com/news/02d45650-b569-4d12-b44d-8d6d8aeb6a0d https://finance.biggo.com/news/02d45650-b569-4d12-b44d-8d6d8...
- NitpickLawyer 3mo agoAgreed. My (somewhat educated) guess is that top labs have healthy margins on API pricing. But this release will add another 3rd party / clear of conflict datapoint in this estimation.
- knollimar 3mo agoThat numner blends in training or no?
- npn 3mo agoeven deepseek, with their current (dirt cheap) price, can earn enough profit to cover the cost (hardware investment?) in 10 months.
- likium 3mo agoWill the model even be competitive in 10 months though? Seems like models that reach top 20 on OpenRouter see 50% of all token spend by day 80, and 80% by day 180.
- entrope 3mo agoAs long as the hardware can be used on newer models, hardware costs can be recouped running a future model. But if they're hoping to recoup non-recurring engineering costs rather than just hardware costs, they do need to consider the useful lifetime of the specific model.
- woctordho 3mo agoSpeaking of finetune, currently a common practice is LoRA over bnb 4-bit base model, but I think it's time to replace bnb with GGUF as the base model format. GGUF is actively supporting new model architectures and more aggressive quantizations. I've made some proof of concept in https://github.com/woct0rdho/transformers5-qwen3.5-recipe https://github.com/woct0rdho/transformers5-qwen3.5-recipe . We can finetune Qwen3.5-35B-A3B in 16 GiB VRAM, and DeepSeek-V4-Flash (284B-A13B) in 90 GiB VRAM, without CPU offload. This works well on unified memory machines like Strix Halo. Even so, larger models like Kimi-K3 still require multiple GPUs and nodes, and there are a lot more to do compare to single-GPU training.
- maxignol 3mo agoI dont quite understand why GGUF is better optimized. Are the performances better for the same amount of VRAM ?
- woctordho 3mo agoGGUF is at least better than bnb. From what I know, bnb does not yet find a way to quantize MoE with enough accuracy, and maintain the dequant-MoE kernels. In the age of Qwen 3.0, people tried to make some bnb '4-bit' quants of MoE models, but actually the MoE part is not quantized. It's a pity that even Unsloth gave up low-VRAM finetuning with MoE (although they're making their GGUFs for inference), and the world of local training looks stagnated for months. GGUF is maintained by all the llama.cpp developers. There are many quantization formats and algorithms under this container format, some are optimized for MoE (such as APEX quant), some for CPU and some for GPU, some work surprisingly well below 4-bit (and even near 1-bit). It also supports recent architectures like linear attentions and mHC.
- qeternity 3mo agoI cannot be sure what the likes of Cursor have done, but I think it's incredibly unlikely that they have trained a QLoRA for Composer. It's almost certainly full parameter post training of the original model weights.
- vb-8448 3mo ago> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing". No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)
- amelius 3mo ago> Without training cost you can infer only the marginal cost of serving this kind of models. Which is by far the most interesting number of the two. > Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?) If you get close in output quality, then does that matter?
- embedding-shape 3mo ago> If you get close in output quality, then does that matter? When you're trying to estimate/infer the costs of serving the tokens and even include the cost of training the weights in order to output tokens then yeah, why wouldn't that matter?
- jvuygbbkuurx 3mo agoThat only matters if you are an investor not if you are a consumer.
- embedding-shape 3mo agoWell, or if you're participating in a discussion on HN where the sub-topic happens to be "if labs are subsidising tokens on API pricing" and literally the cost of serving the tokens is relevant to the sub-topic people are trying to discuss...
- solenoid0937 3mo agoConsumers want better models too, of course it matters
- walrus01 3mo agoIt will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge price difference in what you can do with buying a used 4U rackmount server and putting 3TB of RAM in it (64GB DIMMs x quantity 32 in a quad socket xeon, you can see some benchmark prices on eBay for sets of 16 or 32 matched 64GB ECC DIMMs) for <$30,000, vs the cost of trying to run it on real GPU hardware. Now obviously, as of the time I write this, the full precision hasn't been released nor has anyone like unsloth run it through quantization yet to produce a "Q8" or "Q8-XL" variant of it. But I think it's going to need more than 1536GB of RAM, with a usable and large amount of context, more like 2TB and preferably 2.5 to 3TB. I also predict that people who try to run it in Q4 and Q6 will get the worst of both worlds, less precision/lost knowledge but also not reliable output that comes out too slow. In my personal opinion if I'm going to deal with something that is smart but slow and running on limited budget hardware, I need it to be Q8.
- sandworm101 3mo agoThose old LTT videos of high core-count threadrippers running GPU benchmarks become more relevant each day.
- walrus01 3mo agoThe performance bottleneck is not really so much the number of cores or processing power in each core, but the memory bus bandwidth to/from the CPU. I have an older dual socket xeon server here which is a CPU-only LLM test machine with 256GB of RAM and the actual CPU stress is not much, I can even quantify this by how little it spins up the CPU fans to meet thermal load (the CPUs are operating at nowhere near their 180W per socket max capacity, compared to like, crunching prime numbers or running cpuburn). But the memory bus speed is fully committed when generating tokens or thinking.
- 3mo ago
- woadwarrior01 3mo agoSince the model is natively MXFP4, I think it'll be even more interesting on the hardware front. It'll comfortably fit on a 8x AMD MI355X node. I suspect that'll drive token prices down, further.
- deleted 3mo ago[deleted]
- Der_Einzige 3mo agoAnyone who thinks that the labs are not profitable on per token API pricing is delusional and hilariously wrong.
- flaburgan 3mo agoIt all depends if you count the fixed cost of training or not. And the cost of the hardware.
- flyinglizard 3mo agoOr even the basis of the cost of hardware. There are lease deals, capacity traded for equity, various programs by Nvidia, there's absolutely massive depreciation, etc.
- Der_Einzige 3mo agoDepreciation is massively overrated by Micheal Berry and his ilk. Everyone keeps thinking those A100s only have 6 more months of life, and yet they're still going for more than they did per hour in 2024. Show me evidence that A100 prices have collapsed, and maybe GPU depreciation will be relevant to the market.
- rhdunn 3mo agoIf it is a mixture of experts (MoE) model like the 2.x models, won't this reduce the hardware needed to run the model? The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. On a single B200 you can have 5-6 experts loaded into VRAM at a time. Realistically that would be 3-4 to account for the context. [!] [!] With this and other MoE models it looks like an interesting area for research would be to detect or predict which models would be needed ahead of time. That way you could schedule the load into VRAM step before the weights are needed. That way you shouldn't lose much/any performance from offloading the weights to RAM.
- embedding-shape 3mo ago> The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090 Without leveraging system RAM and/or SSDs, I don't think you can, or how exactly are you running this, if this is something you are doing today? With CPU/expert offloading you could probably do it with a 5090 + 1TB of RAM or something like that, but absolutely not on a single 5090 entirely within VRAM.
- rhdunn 3mo agoYes, that's what I was saying w.r.t. expert offloading, i.e. ensuring that the GPU could fit the active parameters not all the parameters.
- embedding-shape 3mo agoAlright, I guess I misunderstood. To be fair, this part: > The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. Is painting a very different perspective, even considering the latter parts it's hard to read that as "Of course offloading everything else that doesn't fit on the GPU itself". But anyways, it's been clarified now so no harm :)
- robotswantdata 3mo ago
- vorticalbox 3mo agoI think cursor will likely do a grok fine tune rather than a kimi one for the next composer. they noted in their blog post they didn't focus purely on coding for grok 4.5.
- NitpickLawyer 3mo agoMost likely, since they were acquired. But for us outsiders it would be a cool thing, to see if the delta is the same between kimi2.x + cursor data -> kimi3 + cursor data.
- swingboy 3mo agoSay a single Kimi K3 is deployed on 16 x B200s: how many concurrent users can that handle? I realize the question assumes a major simplification that everyone's prompts/sessions are the same.
- serf 3mo ago>I realize the question assumes a major simplification that everyone's prompts/sessions are the same. well, exactly. that's tough to answer without just average sampling because some users will ask the model "what's todays date" or "what color is the sky?" and some users will ask "Let's rewrite the linux kernel in brainfuck."
- Pxtl 3mo agoSo maybe the better question would be how many concurrent actively-thinking/working agents it could handle?
- aae42 3mo agoIn my admittedly little experience running a tiny LLM service on a single 5090 for friends, I would guess a fair bit. It depends on how many tok/sec you're targeting and how much memory you're allowing be used for context (and their context). Definitely more than 1, MAYBE an order of magnitude.
- blks 3mo agoIf it takes so much resource to run, how does the sharing of a single llm works? There is some interface that basically submits context/cache plus current promt, from each user, doing basically time-sharing compute?
- mnahkies 3mo agoI believe they batch requests so the same weights in vram are shared across many requests https://www.baseten.co/blog/continuous-vs-dynamic-batching-for-ai-inference/ https://www.baseten.co/blog/continuous-vs-dynamic-batching-f...
- withinboredom 3mo agoWe have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy -- it cannot be compressed further. On all tested large models, it's a ratio of 1.34-1.23 -- and smaller models up to 3.76x. It also increases the effective bandwidth by the same rate.
- kaxu-64 3mo ago[flagged]
- withinboredom 3mo agoThanks for your opinion
- Alpha3031 3mo agoThat's very interesting. Does that mean you can reduce say, a 30B class Q8 from ~30 GB down to 10 GB or less?
- withinboredom 3mo ago704gb -> 564gb; 358 gb -> 270 gb; 28.79 gb -> 7.65 gb; 439 gb -> 93 gb It depends on the total entropy of the model. Smaller models have less entropy.
- johndough 3mo ago> Smaller models have less entropy. Interesting. Why is that? I would have expected the opposite, since larger models have to try less hard to fit the training data. Or maybe this leaves more parameters with random initialization, resulting in higher entropy for larger models?
- withinboredom 3mo agoI honestly don't know... I didn't train the models, so I can't tell why the math works out that way. It just does. I suspect it has to do with the fact that all the small models I've tested have been quantized. I don't know of any small model trained from scratch. If you know of any, I'd be happy to encode it and see what it looks like.
- Palmik 3mo agoYou're assuming inference providers are going to sell tokens at cost. You're also assuming that the inference providers have will optimized inference engine. I haven't seen that to be the case so far, to be honest. Take a look at GLM 5 vs GLM 5.2 pricing -- GLM 5.2 cost more despite being the same model. Take a look a look at DeepSeek, which hosts DS v4, profitably, yet others aren't able or willing to match the price.
- nl 3mo agoI think it's unclear the the DS hosted prices are profitable. AFAIK that haven't claimed that. OTOH, the multiple providers who have settled around the same price point ($3.48/M output tokens for multiple providers with good reputations) does indicate where it is profitable: https://openrouter.ai/deepseek/deepseek-v4-pro#providers https://openrouter.ai/deepseek/deepseek-v4-pro#providers
- vdomi 3mo agoDeepseek has claimed x6 profit on inference
- NitpickLawyer 3mo agoI'll be honest, I typed that message while having morning coffee, so it's just a quick reaction from my part, not a heavily researched article in a journal :) But I do think that the median price where this settles will tell us something about the floor at which it is profitable to serve this model. > DeepSeek, which hosts DS v4, profitably I specifically mentioned 3rd party providers, because there can be an argument that model creators themselves are subsidising tokens to gather training data for the next model. In fact, ds are public about their gathering of data (at least on openrouter they're marked as such). So that 0.x price point for dsv4-pro is likely subsidised.
- Palmik 3mo agoBased on the best available information, DeepSeek is pricing the API such that they can repay their infra capex over 10 months, while deprecating/amortizing the cost of said infra over 3 years. For my product, I run GLM 5.2 and other models myself, in production, on rented hardware. Paying API prices would cost much more. EDIT: You can now see several other third-party providers for Kimi K3 (Nebius, Fireworks). All charge exactly the same as the first-party. Does that mean that their costs are the same? Seems quite unlikely. It's simply not an efficient market, yet.
- narwhster 3mo agoThis is a very insightful eye opening take. I haven't even thought of it this way. This really is the first open model to be as big as the frontier has been until now. I think this release is actually great both ways when you think about it. We gonna be able to learn knowledge that labs have been hiding from us (e.g. cost like you mentioned). And labs could learn from whatever optimization techniques people come up with when trying to host this model. It's honestly just good for everyone in my opinion.
- XCSme 3mo agoSo nowadays the hardware and hosting providers must be in an optimization race, whoever can make the model just a bit smaller or more efficient (to fit on fewer/less powerful cards) will have a huge advantage and can make a lot of money. I am curios what's the most profitable thing to "plant" (agriculture analogy) on the land (cards) that you have have: web hosting, vps, llms, image/video generation, etc
- epolanski 3mo ago> The latest AISI benchmarks on cybersec place it above glm5.2, but still way way behind SotA closed models. Sota closed models don't even answer cybersecurity questions lol.
- stellamariesays 3mo ago[flagged]
- dominotw 3mo agopardon my ignorance but is fine tuning still considered viable in face of rapid model releases. Is it really worth it? my friends whove tried in their companies gave up on it.
- stackghost 3mo ago>realistically you'll need 16x for context / throughput optimisation Sounds like I'm buying a lottery ticket this week so I can drop $800k on hardware.
- kiratp 3mo agoAISI is capped at 100M tokens and K3 is less token efficient than Anthropic/OpenAI models. There is an argument to be made, looking at AISI results, that with uncapped tokens it would be just slightly behind the closed weight players.
- JacobAsmuth 3mo agoIt's 3/15 - https://openrouter.ai/moonshotai/kimi-k3 https://openrouter.ai/moonshotai/kimi-k3 If you're going to open source your model, why would you set your own price high enough that other providers could easily and profitably undercut you?
- deleted 3mo ago[deleted]
- nullbio 3mo ago"Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing"." - The thing is though, that Anthropic and OpenAI have breakthroughs in optimization that none of the Chinese labs have. So it'll demonstrate a ceiling, but not a floor.