3 ms·
Yup, I experience the same. I don't know what they do to achieve this but it gives them this edge, really curious to learn more about what makes it so good at i
by Alifatisk 9mo ago
Yup, I experience the same. I don't know what they do to achieve this but it gives them this edge, really curious to learn more about what makes it so good at it.
- in-silico 9mo agoA lot of people point to the Muon optimizer that Moonshot (the creators of Kimi) pioneered. Compared to the standard optimizer AdamW, Muon amplifies low-magnitude gradient directions which makes the model learn faster (and maybe gives Kimi its unique qualities). Muon paper: https://arxiv.org/abs/2502.16982 https://arxiv.org/abs/2502.16982
- Alifatisk 9mo agoWow! Thank you