3 ms·
You might be interested in https://datasette.io/plugins/datasette-faiss https://datasette.io/plugins/datasette-faiss, which I'm using alongside openai-to-sqlite
by tomd 4y ago
You might be interested in https://datasette.io/plugins/datasette-faiss https://datasette.io/plugins/datasette-faiss, which I'm using alongside openai-to-sqlite for similarity search of embeddings, following @simonw's excellent instructions at https://simonwillison.net/2023/Jan/13/semantic-search-answers/ https://simonwillison.net/2023/Jan/13/semantic-search-answer...
- uh_uh 4y agoThanks, but the index being in-memory makes it unsuitable for large data sets :/
- iandanforth 4y agoCan you say more? Usually projects that gravitate to SQLlite are not those that require massive scale and a FAISS index of a few GB covers a lot of documents.
- sharemywin 4y agoyou might check out some vector databases: https://milvus.io/ https://milvus.io/ AND pinecone.io there are others too
- uh_uh 4y agoMy dataset is going to be around 10M documents. With OpenAI embeddings, that will be around 62GB. AFAIK SQLite should be able to handle that size, but I haven't tried. This is not going to be my primary DB. I would update this maybe once a day and the update doesn't have to be super fast.
- simonw 4y agoThere is a way of running disk-backed FAISS indexed that don't all fit in memory but I've not quite figured out how to do that yet: https://github.com/facebookresearch/faiss/issues/2675 https://github.com/facebookresearch/faiss/issues/2675
- jamesblonde 4y agoOpenSearch K-NN plugin supports FAISS and it's disk based: https://opensearch.org/docs/latest/search-plugins/knn/index/ https://opensearch.org/docs/latest/search-plugins/knn/index/
- deleted 4y ago[deleted]
- uh_uh 4y agoOpenSearch looks like the best so far, all my requirements combined!