3 ms·

So they took a NN taking embeddings as input, added a first layer going from one-hot encoding to embeddings (you just use the embedding matrix as first layer),

So they took a NN taking embeddings as input, added a first layer going from one-hot encoding to embeddings (you just use the embedding matrix as first layer), did a SVD on it and retrained the network on top. Using a SVD+retraining to compress a neural network is a common idea, their gain come mostly from the idea of applying that to the embeddings which are usually considered separately from the network (but do take space on your device).

anishacharya 8y ago
The idea ismore like : start from a full embedding layer train the network, after the training stabilizes break the first layer into two using the SVD formulation in an online fashion and continue training - after only a few more epochs you gain back all accuracy almost. This is particulary interesting and efficient compared to quantization / pruning methods as there is no way to regain lost accuracy in those methods... quantized distillation can be an option but too slow as pointed out by the authors, offline embedding compression like FastText.zip compresses embedding matrix offline and seems to be less information preserving than this method leading to worse performance. Another important thing to note: this method shows significant improvement over quantization/ pruning/ hashing when the network is deep. Deeper the network more is the quantization error due to more FLOPs in low precision - whereas this algorithm being principled from an information theoretic perspective maintains steady performance irrespective of network depth actually deeper nets will give better gains. In many of our experiments on smaller tasks this method actually acts as a cheap regularization as well leading to boosted performance. Also i guess we need to appreciate the mathematical elegance and ease of implementation. Also look at the Numerical Analysis that guarantees and provides bounds compared to Quantization methods.

anishacharya 8y ago

The idea ismore like : start from a full embedding layer train the network, after the training stabilizes break the first layer into two using the SVD formulation in an online fashion and continue training - after only a few more epochs you gain back all accuracy almost. This is particulary interesting and efficient compared to quantization / pruning methods as there is no way to regain lost accuracy in those methods... quantized distillation can be an option but too slow as pointed out by the authors, offline embedding compression like FastText.zip compresses embedding matrix offline and seems to be less information preserving than this method leading to worse performance. Another important thing to note: this method shows significant improvement over quantization/ pruning/ hashing when the network is deep. Deeper the network more is the quantization error due to more FLOPs in low precision - whereas this algorithm being principled from an information theoretic perspective maintains steady performance irrespective of network depth actually deeper nets will give better gains. In many of our experiments on smaller tasks this method actually acts as a cheap regularization as well leading to boosted performance. Also i guess we need to appreciate the mathematical elegance and ease of implementation. Also look at the Numerical Analysis that guarantees and provides bounds compared to Quantization methods.