4 ms·
Yes, DistilBERT https://arxiv.org/abs/1910.01108 https://arxiv.org/abs/1910.01108 is in fact the closest case I know of. But it is too small (distilling from 11
by sanxiyn 3y ago
Yes, DistilBERT https://arxiv.org/abs/1910.01108 https://arxiv.org/abs/1910.01108 is in fact the closest case I know of. But it is too small (distilling from 110M to 66M) and both BERT and DistilBERT is intended to be used (and benchmarked) with separate fine tuning for specific tasks, so they are not general.