4 ms·
Training these quantized word vectors has to be done in full precision (so no memory gains during training the word vectors). But when you save them to disk ev
by maxlam 9y ago
Training these quantized word vectors has to be done in full precision (so no memory gains during training the word vectors).
But when you save them to disk every value is either -1/3 or +1/3 so one could encode the word vectors in binary. This can lead to reducing memory usage during application time if you kept the word vectors in this compressed format (though you'd need to write a decode function in tensorflow or pytorch to take a sequence of bits corresponding to a word and convert it into a vector of -1/3s and +1/3s)
- opportune 9y agoOh interesting I see, so this is like a digital format mostly for sharing models between storage / over networks. I definitely think it would be possible (and useful!) to extend it to in-memory usage, though a C-function wrapper might be better than a native python function. Personally I'm often more frustrated by word2vec's size in memory than in storage so it might be used more in this manner. Would you mind if I submit a pull request?
- maxlam 9y agoAbsolutely, please do!