4 ms·
Note that LMDB uses mmap already, and has the same advantages you list. (no ser/deser, OS controls cache).
by hyc_symas 8y ago
Note that LMDB uses mmap already, and has the same advantages you list. (no ser/deser, OS controls cache).
- danieldk 8y agoBut the embedding matrix will not be stored contiguously, which has benefits, especially when frequent words are stored together (which happens in some implementations, because they sort the vocabulary by frequency), due to page-granularity of mapping, etc. Also, in this particular implementation: By default, LMDB Embeddings uses pickle to serialize the vectors to bytes (optimized and pickled with the highest available protocol). However, it is very easy to use an alternative approach such as msgpack So, they use serialization/deserialization.