4 ms·
Note that the original word2vec binary format is extremely bad for low-memory use. It stores the words and vectors interleaved (and words are obviously of a var
by danieldk 8y ago
Note that the original word2vec binary format is extremely bad for low-memory use. It stores the words and vectors interleaved (and words are obviously of a variable length). Of course, you could circumvent this problem by building a separate (sorted) index file.
However, newer formats, such as the fastText binary format, store the embedding matrix contiguously. In such formats you can just memory-map the embedding matrix and you only have to load the vocabulary into memory. This is even simpler than the approach described here and in the Delft README, you don't have any serialization/deserialization overhead [1], you can let the OS decide how much to cache in memory, and you have one dependency less (mmap is in POSIX [2]).
[1] Of course, if you have a system with different endianness, you have to do byte swapping.
[2] http://pubs.opengroup.org/onlinepubs/7908799/xsh/mmap.html http://pubs.opengroup.org/onlinepubs/7908799/xsh/mmap.html
- hyc_symas 8y agoNote that LMDB uses mmap already, and has the same advantages you list. (no ser/deser, OS controls cache).
- danieldk 8y agoBut the embedding matrix will not be stored contiguously, which has benefits, especially when frequent words are stored together (which happens in some implementations, because they sort the vocabulary by frequency), due to page-granularity of mapping, etc. Also, in this particular implementation: By default, LMDB Embeddings uses pickle to serialize the vectors to bytes (optimized and pickled with the highest available protocol). However, it is very easy to use an alternative approach such as msgpack So, they use serialization/deserialization.
- anewhnaccount2 8y agoNote also: https://radimrehurek.com/gensim/models/keyedvectors.html https://radimrehurek.com/gensim/models/keyedvectors.html
- Radim 8y agoNote that like the article says, memory-mapping (mmap) vectors from disk already works in Gensim: https://radimrehurek.com/gensim/models/keyedvectors.html https://radimrehurek.com/gensim/models/keyedvectors.html The low memory footprint in LMDB is interesting. There were also some exciting papers recently that used compression (vector quantization) [0]. We're still evaluating whether to add those to Gensim [1]. [0] word2bits, https://twitter.com/RadimRehurek/status/976015607941517312 https://twitter.com/RadimRehurek/status/976015607941517312 [1] word2bits PR, https://github.com/RaRe-Technologies/gensim/pull/2011#issuecomment-380071273 https://github.com/RaRe-Technologies/gensim/pull/2011#issuec...