3 ms·
This is really cool! Had a question on the MoE: in kimmi-K2/model.py, the router does torch.topk(..., k=self.num_experts) while every other MoE uses k=self.top
by tweedler290 14d ago
This is really cool!
Had a question on the MoE: in kimmi-K2/model.py, the router does torch.topk(..., k=self.num_experts) while every other MoE uses k=self.top_k. The ctor's top_k=8 is never stored either.. Is that intentional dense routing or should that be self.top_k?
- anuj0456 14d agothanks for pointing out. its a bug
- tweedler290 13d agour most welcome! if you're curious how I found it, I have made a code intelligence system. I literally fed it your repo and asked for bugs. You can try it yourself here: https://benzi.fly.dev/ https://benzi.fly.dev/
- anuj0456 14d agofixed