4 ms·
hnswlib (https://github.com/nmslib/hnswlib https://github.com/nmslib/hnswlib) is a strong alternative to faiss that I have enjoyed using for multiple projects.
by wskish 4y ago
hnswlib (https://github.com/nmslib/hnswlib https://github.com/nmslib/hnswlib) is a strong alternative to faiss that I have enjoyed using for multiple projects. It is simple and has great performance on CPU.
After working through several projects that utilized local hnswlib and different databases for text and vector persistence, I integrated hnswlib with sqlite to create an embedded vector search engine that can easily scale up to millions of embeddings. For self-hosted situations of under 10M embeddings and less than insane throughput I think this combo is hard to beat.
https://github.com/jiggy-ai/hnsqlite https://github.com/jiggy-ai/hnsqlite
- antman 4y agoI looked a bit and the code, I think it would be kow hanging fruit to add additional sqlite fields except the vector ones. Even if any filtering happens kind of suboptimally in post processing.
- leobg 4y agohnswlib supports pre-filtering
- antman 4y agoThanks Leobg! For anyone else: you pass it directly in metadata see https://github.com/jiggy-ai/hnsqlite/blob/main/test/test_collection_filter.py#L7 https://github.com/jiggy-ai/hnsqlite/blob/main/test/test_col... https://github.com/jiggy-ai/hnsqlite/blob/main/test/test_collection_filter.py#L41 https://github.com/jiggy-ai/hnsqlite/blob/main/test/test_col...
- Labo333 4y agoI totally agree and hnswlib is actually much faster than FAISS on CPU. I'm really happy to see `hnswlib` as a Python dependency since I'm the one who implemented PyPI support: https://github.com/nmslib/hnswlib/pull/140 https://github.com/nmslib/hnswlib/pull/140
- fzysingularity 4y agoInteresting - is there a good reference to back this claim? Curious to hear what overheads Faiss would have if it's configured with similar parameters to build the HNSW graphs. Is that what you A/B-tested in practice?
- wskish 4y agohttp://ann-benchmarks.com http://ann-benchmarks.com hnswlib implementation of hnsw is faster than faiss's implementation. Faiss has other index methods that are faster in some cases, but more complex as well.
- wskish 4y agoThank you for this! This project is really hnswlib-sqlite just shortened into hns(w)qlite.
- nl 4y agoI like Faiss but I tried Spotify's annoy[1] for a recent project and was pretty impressed. Since lots of people don't seem to understand how useful these embedding libraries are here's an example. I built a thing that indexes bouldering and climbing competition videos, then builds an embedding of the climber's body position per frame. I then can automatically match different climbers on the same problem. It works pretty well. Since the body positions are 3D it works reasonably well across camera angles. The biggest problem is getting the embedding right. I simplified it a lot above because I actually need to embed the problem shape itself because otherwise it matches too well: you get frames of people in identical positions but on different problems! [1] https://github.com/spotify/annoy https://github.com/spotify/annoy