4 ms·
Quick "ask HN": I'm currently working on a semantic search solution, and one of the challenges is to be able to query billions of embeddings easily (single-digi
by aforwardslash 3y ago
Quick "ask HN": I'm currently working on a semantic search solution, and one of the challenges is to be able to query billions of embeddings easily (single-digit seconds). I've been testing different approaches with a small dataset (50-100 million embeddings, 512 or 768 dimensions), and all databases I've tried have somewhat severe issues with this volume of data (<100GB of data) on my local machine. I've tried milvus, chroma, clickhouse, pgvector, faiss and probably some others I don't recall right now. Any suggestions on additional databases to try out?
- lulf 3y agoCheck out https://vespa.ai https://vespa.ai
- NeuralJoker 3y agoI've used Elasticsearch and Pinecone before but never locally. Easy to scale, relatively low latency but can get pricey.
- pbourke 3y agoMy team at Microsoft works on DiskANN, which was designed for this kind of use case. https://github.com/microsoft/DiskANN https://github.com/microsoft/DiskANN
- dmezzetti 3y agoYou can try txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) with a Faiss backend. This Faiss wiki article might help (https://github.com/facebookresearch/faiss/wiki/Indexing-1G-vectors#1b-datasets https://github.com/facebookresearch/faiss/wiki/Indexing-1G-v...). For example, a partial Faiss configuration with 4-bit PQ quantization and only using 5% of the data to train an IVF index is shown below. faiss={"components": "IVF,PQ384x4fs", "sample": 0.05}