2 ms·
Knowledge distillation for transformers is already a thing and it is still actively researched since the potential benefits of not having to run these gigantic
by sigmoid10 4y ago
Knowledge distillation for transformers is already a thing and it is still actively researched since the potential benefits of not having to run these gigantic models are enormous.
- lossolo 4y agoIndeed but all the papers I've read about this were only for simple transformers, for now at least.