2 ms·
22,000-wide output sourced by a 4096-wide embedding You will want to use hierarchical outputs in this case. Take a look at Hinton's 'Knowledge Distillation' pa
by p1esk 8y ago
22,000-wide output sourced by a 4096-wide embedding
You will want to use hierarchical outputs in this case. Take a look at Hinton's 'Knowledge Distillation' paper.
- oneshot908 8y agoSure, that's a nice approximation, and it will reduce performance somewhat IMO, it'd be nice to quantify how much, no?