3 ms·
It probably also has to do with their internal infra. If it were just about dense models being easier for the OSS community to use & build on, they should proba
by whimsicalism 2y ago
It probably also has to do with their internal infra. If it were just about dense models being easier for the OSS community to use & build on, they should probably be training MoEs and then distilling to dense.