4 ms·

This is interesting work. We at Plasticity (YC S17) have open sourced something similar called Magnitude (https://github.com/plasticityai/magnitude https://gith

This is interesting work. We at Plasticity (YC S17) have open sourced something similar called Magnitude (https://github.com/plasticityai/magnitude https://github.com/plasticityai/magnitude) a few months ago for querying vector embeddings quickly with low memory usage using SQLite and a standard universal file format (.magnitude) between word2vec, fastText, and GloVe. We also added features for out-of-vocabulary lookups, bulk queries, concatenating embeddings from multiple models, etc. It may also be of interest to folks looking at this. We also have a paper on Magnitude we will be presenting at EMNLP 2018 and ELMo support is coming soon!

plusepsilon 8y ago
Oh wow, how did I not know about this :) Looks super useful. How would ELMo work if a neural network needs to be run?
- patelajay285 8y ago
  Thanks! Still working on it, but it will embed the ELMo model in the .magnitude file, wrap the ELMo model interface code so that it is standardized with Magnitude's features (querying, concatenating, iterating, etc.) and will use ELMo's OOV method natively.
hyc_symas 8y ago
SQLite (and SQL in general) is ridiculously slow for specialized applications like this. http://www.lmdb.tech/bench/microbench/ http://www.lmdb.tech/bench/microbench/
- patelajay285 8y ago
  It depends what you are optimizing for. It's a little slower for random disk reads, but it doesn't matter given we utilize an in-memory LRU cache and most text has a Zipf-ian distribution which will hit the cache very often even with a small cache size. (see my comment elsewhere on this thread) Developers are more familiar with SQLite, it's bundled with Python, it is easy to add to nearly every language, and it has specialized features we use like the Full Text Search module for out of vocabulary lookups.
  - hyc_symas 8y ago
    Wrappers for LMDB are available for nearly every language as well. https://symas.com/lmdb/technical/#wrappers https://symas.com/lmdb/technical/#wrappers
vstuart 8y ago
Note also: * reddit (Storage and retrieval of Word Embeddings in various databases): https://www.reddit.com/r/LanguageTechnology/comments/9cf0zk/storage_and_retrieval_of_word_embeddings_in/ https://www.reddit.com/r/LanguageTechnology/comments/9cf0zk/... * ... links to this GitHub repo: https://github.com/martialblog/word_embedding_storage https://github.com/martialblog/word_embedding_storage
domhudson 8y ago
I have tried plasticity (even submitted a PR*) before putting together this repo but found it horrendously slow in comparison. It really does depend though; as plasticity utilise an in-mem cache if you're querying on many similar words repeatedly, that is likely to be faster. EDIT: sorry it wasn't a PR but an issue

plusepsilon 8y ago

Oh wow, how did I not know about this :) Looks super useful. How would ELMo work if a neural network needs to be run?

patelajay285 8y ago
Thanks! Still working on it, but it will embed the ELMo model in the .magnitude file, wrap the ELMo model interface code so that it is standardized with Magnitude's features (querying, concatenating, iterating, etc.) and will use ELMo's OOV method natively.

hyc_symas 8y ago

SQLite (and SQL in general) is ridiculously slow for specialized applications like this. http://www.lmdb.tech/bench/microbench/ http://www.lmdb.tech/bench/microbench/

patelajay285 8y ago
It depends what you are optimizing for. It's a little slower for random disk reads, but it doesn't matter given we utilize an in-memory LRU cache and most text has a Zipf-ian distribution which will hit the cache very often even with a small cache size. (see my comment elsewhere on this thread) Developers are more familiar with SQLite, it's bundled with Python, it is easy to add to nearly every language, and it has specialized features we use like the Full Text Search module for out of vocabulary lookups.
- hyc_symas 8y ago
  Wrappers for LMDB are available for nearly every language as well. https://symas.com/lmdb/technical/#wrappers https://symas.com/lmdb/technical/#wrappers

vstuart 8y ago

Note also: * reddit (Storage and retrieval of Word Embeddings in various databases): https://www.reddit.com/r/LanguageTechnology/comments/9cf0zk/storage_and_retrieval_of_word_embeddings_in/ https://www.reddit.com/r/LanguageTechnology/comments/9cf0zk/... * ... links to this GitHub repo: https://github.com/martialblog/word_embedding_storage https://github.com/martialblog/word_embedding_storage

domhudson 8y ago

I have tried plasticity (even submitted a PR*) before putting together this repo but found it horrendously slow in comparison. It really does depend though; as plasticity utilise an in-mem cache if you're querying on many similar words repeatedly, that is likely to be faster. EDIT: sorry it wasn't a PR but an issue