6 ms·
Kimi-K3 Technical Report [pdf]
Related: Kimi-K3 on HuggingFace - https://news.ycombinator.com/item?id=49065752 https://news.ycombinator.com/item?id=49065752
- m00dy 3mo agoI would want to see three things before drawing strong conclusions: End-to-end tokens/sec and cost on realistic coding agent trajectories, including tool outputs and retries, not isolated decode benchmarks. Cache hit rates and prefill cost for branching, multi-turn sessions. Router-load distributions after post-training, where expert collapse or specialization problems often show up.
- storus 3mo agoWhat would be the current best method to fine-tune it for my own specific agentic tasks? LoRA + DPO? GRPO? Something else?
- whimsicalism 3mo agoLoRA + SFT, but it'll be big - better to wait for a finetuning API from one of the providers, I wouldn't jump straight to RL or off-policy pseudo-RL like DPO.
- m_ke 3mo agoAlso open sourced a bunch of infra to go with it. Anyone who claims open source and open weights models are "decel" needs to get their head checked https://github.com/MoonshotAI/MoonEP https://github.com/MoonshotAI/MoonEP https://github.com/kvcache-ai/AgentEnv https://github.com/kvcache-ai/AgentEnv https://github.com/MoonshotAI/FlashKDA https://github.com/MoonshotAI/FlashKDA
- jvanderbot 3mo agoThis comment would be much better without the second line
- Der_Einzige 3mo ago[flagged]
- ike_a 3mo agoI'm not sure I understand the case for open-source models being decelerationist, is this it? Decel: - Potentially reduces investor appetite for funding big labs. - More risk of powerful AI getting in bad hands -> more regulation. Accel: - More competition so big labs can't rest on laurels. - More research in open, so all labs can accrete advancements faster. I feel like open-source = acceleration has a much more clear argument. (and how bad would deceleration be in any case?)
- Iolaum 3mo agoOpen Source models decelerate growth of closed AI. For people who think (or want) AI = closed_AI then that argument has weight. Good luck getting them to update their priors.
- StevenWaterman 3mo agoI think it's basically open weights => more inference competition => less profit from inference => less training competition
- f311a 3mo ago> less training competition I think you meant less research and experiments in big labs because they don't get all the AI money. Training is expensive, but they also have more than 10 000 of employees combined and they cost a lot of money.
- zozbot234 3mo agoOpen source AI is actually a lot less "powerful" than genuine frontier models, i.e. it has a much tighter inherent capability ceiling. This is "decelerationist" from a purely AGI-pilled point of view but it's actually great if you're worried about a capabilities arms race putting AI Safety at severe risk. Kimi K3 is plausibly a lot less dangerous than a totally jailbroken ChatGPT/Gemini/Claude Sonnet (let alone Opus or Fable!) and it's quite deeply weird how no one seems to be calling for those models to be banned or restrained by further regulation. Why the double standard against the less concerning (but more efficient!) open weight models?
- deleted 3mo ago[deleted]
- vanuatu 3mo agoTo me its clear that it is decel the only reason other labs can catch up is because the frontier labs can be distilled, and they siphon a % of the labs' revenue to reinvest into the next iteration full accel would mean nationalizing the big 2 labs and locking in manhattan project style until RSI (Edit: some great counterpoints in the replies. my view has definitely been changed!)
- m_ke 3mo agoonly if you only get your news from main stream business press and Big Lab propaganda channels There's no chance K3 is a distill of Fable, it came out way too soon after the limited fable release to be feasbile. If you look at all of the top ML conferences, chinese labs contribute way more to advances in ML than "Open"AI and Anthropic: https://www.reddit.com/r/TheMachineGod/comments/1pi4q7f/papers_at_neurips_2025/ https://www.reddit.com/r/TheMachineGod/comments/1pi4q7f/pape... This K3 release just helped every other lab on the planet stay in the race by making it possible for them to build on top of it, placing them at the frontier starting line instead of having to spend billions of their own dollars and risking it all to attempt to catch up. The open source contributions I linked to above will move the whole field forward and reduce the costs of training and inference for everyone. Open science compounds on it self, every new advancement pushes the field forwards and opens up new grounds for future improvements. It is impossible for a single closed lab to consistently stay ahead of the rest of the field, especially in a huge growing research area like machine learning. The only only advantage the big labs have is money, but the naive scaling game is not sustainable long term when you have to pay 10-100x more then the fast followers and we start getting more and more open models or use case specific models that can handle 90% of high volume use cases. Research is a high variance, low expected value activity, meaning that the few large concentrated labs have to be conservative with their bets and double down on proven things when scaling up. The rest of the field is like a diversified portfolio, with thousands of players making smaller riskier bets that only require a few of them to succeed (like K3 did here, and DeepSeek a year ago) EDIT: also if you look at most of the work from OpenAI, it's mostly taking existing promising open research work and scaling it up. (except for things like CLIP and etc from Alec Radford)
- 3mo ago
- samxli 3mo ago[flagged]
- fahrradflucht 3mo agoLicense: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE > If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose. + the existing 100 million monthly active users, or more than 20 million US dollars for commercial products have to name Kimi clause
- throwaway27448 3mo agoI wonder how they'll figure out who to target for litigation when this license is violated.
- ffsm8 3mo agoThey haven't litigated the last public non-compliance... Despite that one being extremely public. So probably not at all for now.
- Iolaum 3mo agoMaybe because a non public agreement was in place?
- JumpCrisscross 3mo ago> They haven't litigated the last public non-compliance What was it?
- deleted 3mo ago[deleted]
- drawnwren 3mo agoCursor was thought to be but they were later found to be using an authorized provider [1] 1 - https://x.com/Kimi_Moonshot/status/2035074972943831491?lang=en https://x.com/Kimi_Moonshot/status/2035074972943831491?lang=...
- whimsicalism 3mo agoIt's funny that we've finally returned to tanh activation functions, time is a circle.
- pinkmuffinere 3mo agoWow this is fascinating enough that I’m actually going to read tfa lol
- rhdunn 3mo agoFrom the paper (page 6 with a comparison to GLU and SwiGLU) they are not using tanh directly (i.e. f(x) = tanh(x)) but: f_gate(b,x) = b * tanh(x / b) * sigmoid(x) f_up(b,x) = b * tanh(x / b) Looking at the graph I wonder if this is to try and get the best of both GLU (better representation at higher values of x >~ 5) and SwiGLU (the value bump just before 0).
- whimsicalism 3mo agoStill more of a tanh than I've seen in years
- lostmsu 3mo agoWhich part of that is doing gating? I thought gate generally looks like f_gate(x, y) = f(x) * y for some f. In your case b is a constant hyperparameter though, e.g. f_gate(x) = 4 * tanh(x / 4) * sigmoid(x) So where's the gate?
- rhdunn 3mo agoIn the linked "Kimi-K3 Technical Report [pdf]" paper, section 2.3 (Stable LatentMoE, p6) has the table with those equations on (top of p7, using β_1 for the gate branch and β_2 for the up branch). They talk specifically about the function in section 2.3.2 (Sigmoid Tanh Unit GLU, bottom of p7). Figure 2 (p3) has the architecture diagram for K3 (which is the same one on the Kimi K3 blog post https://www.kimi.com/blog/kimi-k3 https://www.kimi.com/blog/kimi-k3). AFAICT, that diagram along with section 2.3 should answer your question. Note: I'm not familiar with LLM/NN architecture to answer it more precisely than that; I only have a surface level understanding from watching various YouTube channels like 3B1B and Welch Labs on the subject.
- a-dub 3mo agoknowledge graph guided task synthesis. very cool! i have long wondered about the "how do you get good coverage of all the tasks" problem. maybe some interesting theoretical work there around the rate of production of new knowledge itself and various mechanisms (human approaches, mechanistic approaches, etc).
- histiq 3mo ago[flagged]
- eamag 3mo agoCan someone explain what are teachers in Multi-Teacher On-Policy Distillation? I can imagine math, coding and other verifiable domains, but they also have biology? Is it where distillation from bigger models come in?
- porridgeraisin 3mo agoThey're other models yes.
- sosodev 3mo agoThey reference https://thinkingmachines.ai/blog/on-policy-distillation/ https://thinkingmachines.ai/blog/on-policy-distillation/ If I understand correctly, it's distillation via having a teacher model score each of the student's tokens for a problem based on their own probabilities of generating that token at each step in the sequence. The reward/loss is then applied as RL. The multi-teacher bit seems to imply they're distilling from multiple models. It's light on the details, but it seems like it could be part of distilling from frontier/closed models. Provided they calculate the logprobs, which OpenAI seems to allow via API but not Anthropic. Maybe they have a way of estimating the logprobs externally? This method can be used to learn any domain from the teacher. Biology included.
- mike_hearn 3mo agoIt's not that light on the details. I read the paper and they train several different models in parallel over a few different domains and then they distill from their own models to get the final model.
- htrp 3mo ago> which OpenAI seems to allow via API but not Anthropic Does openai still allow logprobs in their current gen models?
- colesantiago 3mo agoThis is amazing to witness. Moonshot open sourcing Kimi K3, a frontier AI and other components really means we are getting abundant AI for all of humanity. Kudos to Moonshot for truly being what OpenAI should have been. Fable-level and frontier AI should be open source and available to everyone for free.
- embedding-shape 3mo ago> Kudos to Moonshot for truly being what OpenAI should have been. Kudos to Moonshot for making these weights available for download. Lets not fool ourselves and claim these are "open source" by any understanding of the concept though, there are usage restrictions (even if you download them) and also training data isn't clearly broken down either, nor it it actually using a FOSS license.
- lossolo 3mo ago> Kudos to Moonshot for making these weights available for download. Lets not fool ourselves and claim these are "open source" by any understanding of the concept though, there are usage restrictions (even if you download them) and also training data isn't clearly broken down either, nor it it actually using a FOSS license. They will probably never release the training data because that represents a large part of their competitive moat. The same is true of US companies (Google, OpenAI, Meta etc) none of which has released the full training data for its open models. They use private datasets that cost a lot to acquire, synthetic datasets and a lot copyrighted material for which they don't have licensing. What matters most is that, with the necessary hardware, I can download a near frontier model, run it and modify it however I want. The other concerns you mentioned are just noise. And if my company is generating $20 million in revenue or serving 100 million users, it can probably afford a relatively inexpensive commercial license.
- bilbo0s 3mo ago>They will probably never release the training data because that represents a large part of their competitive moat Not that what you've written isn't the case. However, in addition to what you've written, (or probably even before what you've written), there's the fact that everything they're training on is stolen IP. Same with US LLM labs. Let's not kid ourselve's about where the training data is coming from. They are not asking artists, writers, coders, content creators, etc etc etc for permission to use their creations. Anthropic, Moonshot et al are doing incredible things, but we shouldn't gloss over the costs. Both present and future costs are kind of enormous.
- eamag 3mo ago> we build a self-evolving, hierarchically organized knowledge graph that agents continuously expand through web-scale exploration across knowledge-intensive and coding domains That's interesting!
- kiaansaraiya 3mo ago[dead]
- tudou527 3mo ago[flagged]
- lenerdenator 3mo agoWhat would it take to get an American open model to compete with this?
- boomskats 3mo agoAn act of G̶o̶d̶ Congress?
- chrsw 3mo agoSomething beyond my imagination
- embedding-shape 3mo agoLatest "big" release from any of the bigger American lab must have been GPT-OSS-120b I think? Released ~summer 2025, so pretty much one years ago. Doesn't seem like it'll happen by itself, so something either forcing their hand figuratively, or something forcing their hand literally. Personally I was wishing/hoping for one of the recent Gemma releases to be in the ~100B class at least, but sadly Google is keeping that all for themselves.
- layer8 3mo agoAugust 2025.
- embedding-shape 3mo agoBah, of course, thanks :)
- lossolo 3mo agoYeah, besides that, the only big other open source US model that was worth looking at was Inkling (975B params), Jul 15, 2026. https://thinkingmachines.ai/news/introducing-inkling/ https://thinkingmachines.ai/news/introducing-inkling/
- Stagnant 3mo agoNVIDIA-Nemotron-3-Ultra-550B-A55B was released in June 4th 2026 and I think it was the largest open US model until thinking machines' Inkling (975B) was released a couple of weeks ago.
- deleted 3mo ago[deleted]
- brcmthrowaway 3mo agoIs this p-hacking?
- weberer 3mo agoDoes anyone know if a torrent is available? I think it would take quite a while to download 1.5tb from their servers.
- se4u 3mo agoYour best bet will be to start a network optimized ec2 (or gcp compute instance) and save to some bucket. Cloud network will be a lot more stable than home.
- abratabia 3mo ago[flagged]
- GodelNumbering 3mo agoBack of the envelope calculation (could be off, correct me if I am) If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 parallel agentic workflows (each with ~100k context on average) at ~30 tok/s. Assuming the annual amortization+electricity at $1.5M/year and about 50% average annual utilization, you get less than 60 cents (USD) per million output token, for a frontier model with plenty of capacity to share, all your data never leaving premises and well over an order of magnitude cheaper! As long as a company believes that the openweight models will continue to get more capable and 'AI is here to stay', this model provides the first solid footing for a decision to just buy a rack.
- 999900000999 3mo agoAnd hire 2 or 3 dev ops to keep it running ? That another 400 to 700k. It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.
- clint 3mo agoJust let it manage itself, what could go wrong! :)
- russell_h 3mo ago> However, I don’t trust hosted LLMs for anything that needs to be private. Why not? Do you trust AWS with things that need to be private?
- Kevcmk 3mo agoMore than I trust frontier labs. AWS doesn't need to recoup 9 digits USD of capex
- 3mo ago
- Tejush 3mo agoAny guess on the pre-training tokens/flops they consumed?
- lostmsu 3mo agoTechnical report has a graph vs Kimi K2 with 1e21 FLOPs (but they don't claim that's the entirety of pretraining)
- JacobAsmuth 3mo ago1e21 flops is hilariously wrong. for reference the llama 3 8B model (https://arxiv.org/pdf/2407.21783 https://arxiv.org/pdf/2407.21783) used 10 times that many flops. This model is 350x bigger in total params and 12x bigger in active params and was trained on 3x the data.
- NguyenDat377 3mo agoThat's pretty impressive, I don't want to be the one undermine but I personally think the industry is shifting to rather designs that enable Models to work instead of chugging it all to data and compute ?
- Mohiuddin7 3mo agoThat's the great achievement, but ahh, wanted to try it, but my RAM had other plans
- Mkadri 3mo ago[dead]
- htrp 3mo agoWhy is everyone's pricing the same? licensing agreements with moonshot as an inference provider?
- logicallee 3mo agoWhat are the coding/agent harnesses (like Claude Code or ChatGPT Codex) that can be used with this?
- tesnorindian 3mo ago> Two findings in the Linux kernel illustrate the depth of these results. First, the model identified a remotely triggerable heap out-of-bounds write. The bug was introduced by an incomplete upstream fix and affects all subsequent releases, up to and including the latest upstream code Hope these Linux Kernel bugs due to some incomplete upstream fix identified by K3 patched? Moonshot did not provide much details on this. https://www.opensourceforu.com/2026/07/linux-maintainers-battle-record-ai-fuelled-cve-surge/?utm_source=krypteiasec&utm_medium=email&utm_campaign=daily-threat-brief-july-26-2026 https://www.opensourceforu.com/2026/07/linux-maintainers-bat... Looks like K3 is good with identifying zero days https://www.heise.de/en/news/Kimi-K3-Chinese-AI-finds-several-zero-day-vulnerabilities-in-redis-database-11377430.html https://www.heise.de/en/news/Kimi-K3-Chinese-AI-finds-severa...