3 ms·
I feel there’s a gap missing in this thread (or I may be the one missing it) DeepSeek proved knowledge distillation works very well and cheaply https://en.m.wi
by sghiassy 2y ago
I feel there’s a gap missing in this thread (or I may be the one missing it)
DeepSeek proved knowledge distillation works very well and cheaply https://en.m.wikipedia.org/wiki/Knowledge_distillation https://en.m.wikipedia.org/wiki/Knowledge_distillation
But they didn’t show how to build a new frontier model cheaply.
So, you still need massive investments to build new frontier models. But the bad part, is they can be replicated cheaply
- cubefox 2y agoThis comment seems to be complete nonsense. See here https://arxiv.org/abs/2412.19437v1 https://arxiv.org/abs/2412.19437v1
- gilbetron 2y agoI think you are missing it: https://stratechery.com/2025/deepseek-faq/ https://stratechery.com/2025/deepseek-faq/ That has a great overview - this is a new model, but also a distillation. They used new techniques to make it really cheap (comparatively).
- sghiassy 2y agoThank you for the link. I have a lot of respect for Stratechery. I learned a lot, and agree, I’m the one who was missing it haha