3 ms·
Haven’t read the paper but my guess around that is that the same reason sparse attention networks (where they 0 out many weights) just have the sparse tensors b
by vlovich123 2y ago
Haven’t read the paper but my guess around that is that the same reason sparse attention networks (where they 0 out many weights) just have the sparse tensors be larger.
- mayukhdeb 2y agoIn this paper, we don't zero out the weights. We remove them.
- vlovich123 2y agoThanks for the correction! Can it be retrofitted into existing models through distillation or do you have to train the model from scratch?