3 ms·

I wonder, how does distillation deal with unprobed spaces in the knowledge landscape? Is a distilled model worse in some niche area that was not probed? Presuma

I wonder, how does distillation deal with unprobed spaces in the knowledge landscape? Is a distilled model worse in some niche area that was not probed? Presumably, this is why frontier labs dont distill their own models internally to release them to the public as a servicable frontier model.

tngranados 2mo ago
Do frontier labs not distill their own bigger models into the smaller/cheaper variants? I thought that’s been the case for a while