2 ms·
Very nice work! :) Are there any benchmarks available? I'm curious how this compares to caching frequent word vectors (Zipf's law helps here) and disk-seeking t
by atrudeau 8y ago
Very nice work! :) Are there any benchmarks available? I'm curious how this compares to caching frequent word vectors (Zipf's law helps here) and disk-seeking the rest.
- patelajay285 8y agoSee my top-level reply about a similar library called Magnitude that we have built and open-sourced: https://github.com/plasticityai/magnitude https://github.com/plasticityai/magnitude. It actually utilizes an LRU cache (configurable cache size argument on the constructor) so you can utilize an in-memory LRU cache as you query words off-disk using SQLite indexed disk-seeks. And you're right, due to Zipf-ian properties of most text, you can see gains even with a small in-memory cache size :).