3 ms·
that's because they are distilling the frontier models
by jdmoreira 9mo ago
that's because they are distilling the frontier models
- epolanski 9mo agoFrontier models aren't released, they are closed source. And the Chinese have been a huge source of innovation in the field.
- jdmoreira 9mo agodistilling does not require the models to be released. They simply use the apis. They have been a source of innovation but probably not in training them.
- NoOn3 9mo agoThey just lack of performant hardware. They have enough knowledge. And so they choose a more effective strategy without wasting resources on training from scratch.
- deleted 9mo ago[deleted]
- Tostino 9mo agoYou can do a rough distill through the APIs. You don't need the weights. It was much easier when companies had models on the /completion style APIs, because you could actually get the logits for each generation step, and use that as a dataset to fit your model to. That isn't to diminish the efforts of the Chinese developers though, they are great.
- riku_iki 9mo agois there any evidence that this is significant contribution? My intuition that one need ALOT api credits to distill such large models.