5 ms·

Self-hosting Kimi K3: 20% more hardware cost, 20% better task resolution

161 points by flifenstein 2mo ago

flifenstein 2mo ago
Author here(I'm on the team). We updated this post after Monday's Kimi K3 weights release: fitting the 2.8T model means going from 8×B200 to 8×B300, ~20% more hardware cost, and concurrency drops from 24 to 16 users vs GLM-5.2. Caveat we're upfront about in the post: our 64-task SWEBench Pro subset may be in Kimi's training set, so the 86% resolve rate is an upper bound. Let us know your thoughts, we really value feedback!
- michalpleban 2mo ago
  Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.
  - deflator 2mo ago
    I second this. Anyone considering running their own hardware would also consider quantization since they would have full control over what quant to run. The value proposition changes a lot if you can get 90% of the quality for 50% the price with a quant due to halving your hardware requirement. Thank you for the article, though! Very informative.
  - NitpickLawyer 2mo ago
    Kimi k3 is natively mxfp4 w/ mxfp8 activations. Since they talk about glm 5.2 running on a 8x b200s, that's likely fp8 since it wouldn't fit at bf16 (1.5TB weights alone)
    - johndough 2mo ago
      HuggingFace used an NVFP4 quant of GLM-5.2 to investigate their latest hack, so that might also be worth a try: https://huggingface.co/nvidia/GLM-5.2-NVFP4 https://huggingface.co/nvidia/GLM-5.2-NVFP4
  - mindwraps 2mo ago
    Solid point. We debated multiple optimization methods and decided to do a more vanilla run first for the setups tested. We’re moving on to some more typical optimization methods for our next article… As soon as our teams comes out a well deserved summer break. Thx for the input.
  - walrus01 2mo ago
    So the 'unsloth' Kimi K3 Q8 quantization is about 1.56TB which is close to the original safetensors file in size, and should be nearly lossless. Realistically with enough context you'd need 2TB total RAM whether it's run on a CPU-only platform (like a dual socket xeon with 1024GB connected to each CPU) or some combination of CPU-connected-RAM and GPUs. Unsloth Q4 is 1.51TB which doesn't really help much and isn't likely to be any different in performance. Unsloth Q2-K-XL is 861GB and could possibly fit on 1TB resources but I would want to see a very thorough series of tests to see how much knowledge and capability is lost between it and the full thing. Using an example from the much smaller gemma 4 31B because it's a decent set of charts I could find quickly, I don't know if anyone has published KL divergence charts for Q4 vs Q4 vs Q8 of Kimi K3: https://localbench.substack.com/p/gemma-4-31b-gguf-kl-divergence https://localbench.substack.com/p/gemma-4-31b-gguf-kl-diverg...
- lcampbell 2mo ago
  > Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users) Is that actually the aggregate throughput? 8xB300 (with 4TB/s/GPU bandwidth) is only pushing 8 tg/s/session? That seems… incredibly low, even for an A100B model. Is it actually 122 tg/s per session? (1952 tg/s aggregate throughout)?
Lord-Jobo 2mo ago
>which box to buy … > spark, costs less than a conference trip. I know putting actual prices regionally localizes your article and temporally, with how prices are so unstable. But analysis of “what to buy” without actual prices is borderline meaningless. Overall, good article, very interesting to see a real deployment that’s actually attainable and not just a subscription to a big 3 token plan.
michalpleban 2mo ago
I would love to see such comparisons but with quantized versions, because quantization allows running models on smaller hardware with some quality loss. I am running Qwen3.6-35B-A3B quantized to int4 on an A6000 card that was otherwise just sitting around idle. It works up to a degree, but I would love to see benchmarks comparing different quantizations of several models, especially in quality. This is an important dimension in deciding whether to buy a GPU is worth it, and it is missing from this (otherwise comprehensive) article.
- nicman23 2mo ago
  the a3b does not need to be all in your card. it being a moe you could trivially run it q8 / fp8 if you have the ram
ktosobcy 2mo ago
Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather good (to the point that I use it mostly nowadays)
- infecto 2mo ago
  That Gemma release is what opened my eyes. Even downloading the models on my iPhone was shocking. It reminded me when folks would put Wikipedia on a CD or something similar. Obviously does not compare to a leading model but it’s impressive for something that was running on my phone. I could see thinking token output and it’s directionally interesting thought.
- incognito124 2mo ago
  gemma 31b is my goto day-to-day model. It's adequately capable for coding and natural phenomena, and it follows instructions and skills reasonably well. My main negative trait is all the LLM-isms, but I can get past that for the value it brings
- jszymborski 2mo ago
  On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude. Here are the tok/s I get: - Gemma-4-26B-A4B (Q4_0) = 214 tok/s - Gemma4-31B-QAT (Q4_0) = 58 tok/s - Qwen3.6-35B-A3B (QB_0) = 30 tok/s - Qwen3.6-27B (QB_0) = 9 tok/s EDIT: Updated tok/s after updating llama.cpp
  - Ilaurens 2mo ago
    These token numbers look off for a 5090. For comparison, an rtx 5000 Blackwell SFF with just 470gb/s bandwidth gets me 30 tok/s on gemma4-31B-QAT. Almost 60 tok/s with MTP enabled. A 5090 should get you much more than that!
arjie 2mo ago
This seems like a temporary situation. Utilization maximization is a matter of switching to the software factory unattended flow and intelligently routing tasks that will not require assistance to the unused time. A decade and a half ago we used to run massive map reduce jobs overnight. Code will be handled like this.
joshstrange 2mo ago
I could not focus on the article with all the noise in the background. It was annoying in the header but then it continued down the page. If you want to do this on your marketing pages have at it but for a blog/news style page? Reader mode was the only way to restore sanity.
- ahachete 2mo ago
  Same here. Stopped reading.
- phoghed 2mo ago
  [flagged]
  - joshstrange 2mo ago
    > It should support reduced motion preference though for the more feeble among us. What a strong start to a sentence before veering into a pretty gross equivalency.
  - mindwraps 2mo ago
    Noted. Making sure we have a cleanly readable version asap. We wanted to stand out a bit vs. our mother-brand (imec), but fully agree some of us like a more sober reading experience. Feedback much appreciated.
boomskats 2mo ago
Right, so the 1.4 TB weight footprint straight away makes me wonder how much more important load/unload time becomes in terms of the neocloud product here. With 1.4TB of weights, a fast-enough cold start would basically be _the_ product; fast enough weight-swapping that can make K3 into something you could usefully rent by the hour/workday. Interesting because again, the license Kimi shipped under [0] defines "Model as a Service" as > giving a third party access to language model inference or fine-tuning (e.g., via API) in a manner that allows such third party to exercise meaningful control over the inputs, parameters, or training data and then they have that clause around if you operate such a business above $20M aggregate revenue you need a separate agreement with Moonshot before commercial use, which presumably captures the majority of the larger neoclouds best placed to optimise this. But where's the line? Say you offer infra optimised for GPU inference, warm pools, isolation per customer, exposed control plane, billed per GPU-hour rather than token, the invoice says compute rather than calls. The customer arguably 'self-hosts', you're probably fine? And if you as a provider run the serving stack and hand your customer an inference API, you're inside the definition regardless of whether you charge by the second or by the token. But what about if you give them direct hardware access, but have the weights cached on NVMe / ramdisk hyperlocal to the infra they're renting so that their hf cli pull only takes a few seconds? Sure, a managed warm pool of GPUs with K3 pre-loaded probably isn't ok, but a local hugging face lru cache holding 'whatever your customers pull down most often', superoptimised for fast weight swaps that the customer controls... is? Is it? Again, where's the line? Is it materially different from a local docker registry mirror? What about safetensors checkpoints pre-sharded for the specific hardware topology you're offering? Does it matter whether you perform the checkpoint optimisation yourself and make it available, or merely cache one published on HF that happens to target exactly the hardware you rent out? What if you published that checkpoint yourself? I'm definitely overthinking this, and I'm sure there's been conversation here about this already, but the other kimi threads[1] are enormous. And I am curious. I'm also curious to know whether Moonshot would actually be against a setup like this. Guessing they would if it was AWS (not quite elastic but not that dissimilar), but what about others? Realistically I guess it'd be easier to just talk to them, especially if you were doing it in a way that targets a slice of the pie they never would have gotten anyway due to data residency requirements etc.. [0]: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE [1]:https://news.ycombinator.com/item?id=49065752 https://news.ycombinator.com/item?id=49065752
martinjc 2mo ago
Why does the website have to be so trippy when scrolling? It fucks with my eyes.
matheusmoreira 2mo ago
The costs are so staggering it's becoming clear we're going to be priced out of owning computers altogether. Self-hosting frontier models is a corporation's choice, not an individual's.
- jckahn 2mo ago
  Don't worry, you'll be able to rent your next computer: https://www.apple.com/newsroom/2026/07/apple-upgrade-launches-in-the-united-states/ https://www.apple.com/newsroom/2026/07/apple-upgrade-launche...
  - matheusmoreira 2mo ago
    It's not like people ever owned iPhones to begin with. They're Apple's computers, Apple's just generously allowing their customers to use them, and only on their terms. A monthly iPhone subscription is just the end game. You will nothing, and you'll be happy.
    - Capricorn2481 2mo ago
      Sure, but it's an actual computer I am holding and can use, whereas we are soon to have cloud personal computing.
- tuesdaynight 2mo ago
  If the memory price don't go down anymore, we could say that we are already priced out.
  - matheusmoreira 2mo ago
    The memory prices put computers into the same category as cars. Expensive but still realistic. It's these 200kUSD+ nodes with a dozen or more data center class GPUs that are killing the self-hosting dream. The cheapest nodes are is in the same price category as literal houses. Either we get some competition in this space, or computing will go back to their roots as ivory tower big iron.
walrus01 2mo ago
Quoting from the article: > In our runs, K3 served 16 concurrent sessions (GLM-5.2 managed 24). Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users), and median task time is about 50% longer (38 vs 26 minutes). That makes K3 roughly 8 times slower than our Claude Code baseline. However, K3 makes up for it in quality, resolving 86.4% of tasks, 24 percentage points above both GLM-5.2 and Opus 4.8 (62.5% for both). I think there's some less tangible advantages to self-hosting something on the scale of Kimi K3 that can't be quantified in a specific number like token/s or percentage of problems solved. Such as: a) data privacy/sovereignty from a wide range of possible perspectives, from medical to personal to "we can't have our data go to the USA" for some Canadians and Europeans. b) being able to give it information security/network security tasks and red team scenarios without triggering claude or openai refusals. c) being able to give it information security/network security tasks with zero risk of getting your account banned or investigated by anthropic or openai.
- personjerry 2mo ago
  why are information security / network security tasks bannable?
  - walrus01 2mo ago
    You'd have to ask Anthropic or OpenAI, but I've seen probably more than a dozen reports of it happening. You also risk getting your account banned for discussing certain risky things in chemistry, biochemistry, weapons, conflict, terrorism/history of guerilla warfare. I imagine that within Anthropic and OpenAI there is a huge data set of "big no no!" topics that they maintain and continually add to. For alignment training and for more blunt purposes. For instance ChatGPT 3.5 could be instructed, as satire, to write a press release in the style of the English language propganda arm of ISIS but they put a stop to that fairly quick.
legastenigga 2mo ago
[dead]
Openk3 2mo ago
[flagged]
zop_10 2mo ago
[flagged]

161 points by flifenstein 2mo ago

Author here(I'm on the team). We updated this post after Monday's Kimi K3 weights release: fitting the 2.8T model means going from 8×B200 to 8×B300, ~20% more hardware cost, and concurrency drops from 24 to 16 users vs GLM-5.2. Caveat we're upfront about in the post: our 64-task SWEBench Pro subset may be in Kimi's training set, so the 86% resolve rate is an upper bound. Let us know your thoughts, we really value feedback!

michalpleban 2mo ago
Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.
- deflator 2mo ago
  I second this. Anyone considering running their own hardware would also consider quantization since they would have full control over what quant to run. The value proposition changes a lot if you can get 90% of the quality for 50% the price with a quant due to halving your hardware requirement. Thank you for the article, though! Very informative.
- NitpickLawyer 2mo ago
  Kimi k3 is natively mxfp4 w/ mxfp8 activations. Since they talk about glm 5.2 running on a 8x b200s, that's likely fp8 since it wouldn't fit at bf16 (1.5TB weights alone)
  - johndough 2mo ago
    HuggingFace used an NVFP4 quant of GLM-5.2 to investigate their latest hack, so that might also be worth a try: https://huggingface.co/nvidia/GLM-5.2-NVFP4 https://huggingface.co/nvidia/GLM-5.2-NVFP4
- mindwraps 2mo ago
  Solid point. We debated multiple optimization methods and decided to do a more vanilla run first for the setups tested. We’re moving on to some more typical optimization methods for our next article… As soon as our teams comes out a well deserved summer break. Thx for the input.
- walrus01 2mo ago
  So the 'unsloth' Kimi K3 Q8 quantization is about 1.56TB which is close to the original safetensors file in size, and should be nearly lossless. Realistically with enough context you'd need 2TB total RAM whether it's run on a CPU-only platform (like a dual socket xeon with 1024GB connected to each CPU) or some combination of CPU-connected-RAM and GPUs. Unsloth Q4 is 1.51TB which doesn't really help much and isn't likely to be any different in performance. Unsloth Q2-K-XL is 861GB and could possibly fit on 1TB resources but I would want to see a very thorough series of tests to see how much knowledge and capability is lost between it and the full thing. Using an example from the much smaller gemma 4 31B because it's a decent set of charts I could find quickly, I don't know if anyone has published KL divergence charts for Q4 vs Q4 vs Q8 of Kimi K3: https://localbench.substack.com/p/gemma-4-31b-gguf-kl-divergence https://localbench.substack.com/p/gemma-4-31b-gguf-kl-diverg...
lcampbell 2mo ago
> Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users) Is that actually the aggregate throughput? 8xB300 (with 4TB/s/GPU bandwidth) is only pushing 8 tg/s/session? That seems… incredibly low, even for an A100B model. Is it actually 122 tg/s per session? (1952 tg/s aggregate throughout)?

Lord-Jobo 2mo ago

>which box to buy … > spark, costs less than a conference trip. I know putting actual prices regionally localizes your article and temporally, with how prices are so unstable. But analysis of “what to buy” without actual prices is borderline meaningless. Overall, good article, very interesting to see a real deployment that’s actually attainable and not just a subscription to a big 3 token plan.

michalpleban 2mo ago

I would love to see such comparisons but with quantized versions, because quantization allows running models on smaller hardware with some quality loss. I am running Qwen3.6-35B-A3B quantized to int4 on an A6000 card that was otherwise just sitting around idle. It works up to a degree, but I would love to see benchmarks comparing different quantizations of several models, especially in quality. This is an important dimension in deciding whether to buy a GPU is worth it, and it is missing from this (otherwise comprehensive) article.

nicman23 2mo ago
the a3b does not need to be all in your card. it being a moe you could trivially run it q8 / fp8 if you have the ram

ktosobcy 2mo ago

Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather good (to the point that I use it mostly nowadays)

infecto 2mo ago
That Gemma release is what opened my eyes. Even downloading the models on my iPhone was shocking. It reminded me when folks would put Wikipedia on a CD or something similar. Obviously does not compare to a leading model but it’s impressive for something that was running on my phone. I could see thinking token output and it’s directionally interesting thought.
incognito124 2mo ago
gemma 31b is my goto day-to-day model. It's adequately capable for coding and natural phenomena, and it follows instructions and skills reasonably well. My main negative trait is all the LLM-isms, but I can get past that for the value it brings
jszymborski 2mo ago
On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude. Here are the tok/s I get: - Gemma-4-26B-A4B (Q4_0) = 214 tok/s - Gemma4-31B-QAT (Q4_0) = 58 tok/s - Qwen3.6-35B-A3B (QB_0) = 30 tok/s - Qwen3.6-27B (QB_0) = 9 tok/s EDIT: Updated tok/s after updating llama.cpp
- Ilaurens 2mo ago
  These token numbers look off for a 5090. For comparison, an rtx 5000 Blackwell SFF with just 470gb/s bandwidth gets me 30 tok/s on gemma4-31B-QAT. Almost 60 tok/s with MTP enabled. A 5090 should get you much more than that!

arjie 2mo ago

This seems like a temporary situation. Utilization maximization is a matter of switching to the software factory unattended flow and intelligently routing tasks that will not require assistance to the unused time. A decade and a half ago we used to run massive map reduce jobs overnight. Code will be handled like this.

joshstrange 2mo ago

I could not focus on the article with all the noise in the background. It was annoying in the header but then it continued down the page. If you want to do this on your marketing pages have at it but for a blog/news style page? Reader mode was the only way to restore sanity.

ahachete 2mo ago
Same here. Stopped reading.
phoghed 2mo ago
[flagged]
- joshstrange 2mo ago
  > It should support reduced motion preference though for the more feeble among us. What a strong start to a sentence before veering into a pretty gross equivalency.
- mindwraps 2mo ago
  Noted. Making sure we have a cleanly readable version asap. We wanted to stand out a bit vs. our mother-brand (imec), but fully agree some of us like a more sober reading experience. Feedback much appreciated.

boomskats 2mo ago

Right, so the 1.4 TB weight footprint straight away makes me wonder how much more important load/unload time becomes in terms of the neocloud product here. With 1.4TB of weights, a fast-enough cold start would basically be _the_ product; fast enough weight-swapping that can make K3 into something you could usefully rent by the hour/workday. Interesting because again, the license Kimi shipped under [0] defines "Model as a Service" as > giving a third party access to language model inference or fine-tuning (e.g., via API) in a manner that allows such third party to exercise meaningful control over the inputs, parameters, or training data and then they have that clause around if you operate such a business above $20M aggregate revenue you need a separate agreement with Moonshot before commercial use, which presumably captures the majority of the larger neoclouds best placed to optimise this. But where's the line? Say you offer infra optimised for GPU inference, warm pools, isolation per customer, exposed control plane, billed per GPU-hour rather than token, the invoice says compute rather than calls. The customer arguably 'self-hosts', you're probably fine? And if you as a provider run the serving stack and hand your customer an inference API, you're inside the definition regardless of whether you charge by the second or by the token. But what about if you give them direct hardware access, but have the weights cached on NVMe / ramdisk hyperlocal to the infra they're renting so that their hf cli pull only takes a few seconds? Sure, a managed warm pool of GPUs with K3 pre-loaded probably isn't ok, but a local hugging face lru cache holding 'whatever your customers pull down most often', superoptimised for fast weight swaps that the customer controls... is? Is it? Again, where's the line? Is it materially different from a local docker registry mirror? What about safetensors checkpoints pre-sharded for the specific hardware topology you're offering? Does it matter whether you perform the checkpoint optimisation yourself and make it available, or merely cache one published on HF that happens to target exactly the hardware you rent out? What if you published that checkpoint yourself? I'm definitely overthinking this, and I'm sure there's been conversation here about this already, but the other kimi threads[1] are enormous. And I am curious. I'm also curious to know whether Moonshot would actually be against a setup like this. Guessing they would if it was AWS (not quite elastic but not that dissimilar), but what about others? Realistically I guess it'd be easier to just talk to them, especially if you were doing it in a way that targets a slice of the pie they never would have gotten anyway due to data residency requirements etc.. [0]: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE [1]:https://news.ycombinator.com/item?id=49065752 https://news.ycombinator.com/item?id=49065752

martinjc 2mo ago

Why does the website have to be so trippy when scrolling? It fucks with my eyes.

matheusmoreira 2mo ago

The costs are so staggering it's becoming clear we're going to be priced out of owning computers altogether. Self-hosting frontier models is a corporation's choice, not an individual's.

jckahn 2mo ago
Don't worry, you'll be able to rent your next computer: https://www.apple.com/newsroom/2026/07/apple-upgrade-launches-in-the-united-states/ https://www.apple.com/newsroom/2026/07/apple-upgrade-launche...
- matheusmoreira 2mo ago
  It's not like people ever owned iPhones to begin with. They're Apple's computers, Apple's just generously allowing their customers to use them, and only on their terms. A monthly iPhone subscription is just the end game. You will nothing, and you'll be happy.
  - Capricorn2481 2mo ago
    Sure, but it's an actual computer I am holding and can use, whereas we are soon to have cloud personal computing.
tuesdaynight 2mo ago
If the memory price don't go down anymore, we could say that we are already priced out.
- matheusmoreira 2mo ago
  The memory prices put computers into the same category as cars. Expensive but still realistic. It's these 200kUSD+ nodes with a dozen or more data center class GPUs that are killing the self-hosting dream. The cheapest nodes are is in the same price category as literal houses. Either we get some competition in this space, or computing will go back to their roots as ivory tower big iron.

walrus01 2mo ago

Quoting from the article: > In our runs, K3 served 16 concurrent sessions (GLM-5.2 managed 24). Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users), and median task time is about 50% longer (38 vs 26 minutes). That makes K3 roughly 8 times slower than our Claude Code baseline. However, K3 makes up for it in quality, resolving 86.4% of tasks, 24 percentage points above both GLM-5.2 and Opus 4.8 (62.5% for both). I think there's some less tangible advantages to self-hosting something on the scale of Kimi K3 that can't be quantified in a specific number like token/s or percentage of problems solved. Such as: a) data privacy/sovereignty from a wide range of possible perspectives, from medical to personal to "we can't have our data go to the USA" for some Canadians and Europeans. b) being able to give it information security/network security tasks and red team scenarios without triggering claude or openai refusals. c) being able to give it information security/network security tasks with zero risk of getting your account banned or investigated by anthropic or openai.

personjerry 2mo ago
why are information security / network security tasks bannable?
- walrus01 2mo ago
  You'd have to ask Anthropic or OpenAI, but I've seen probably more than a dozen reports of it happening. You also risk getting your account banned for discussing certain risky things in chemistry, biochemistry, weapons, conflict, terrorism/history of guerilla warfare. I imagine that within Anthropic and OpenAI there is a huge data set of "big no no!" topics that they maintain and continually add to. For alignment training and for more blunt purposes. For instance ChatGPT 3.5 could be instructed, as satire, to write a press release in the style of the English language propganda arm of ISIS but they put a stop to that fairly quick.

legastenigga 2mo ago

[dead]

Openk3 2mo ago

[flagged]

zop_10 2mo ago

[flagged]