5 ms·

Thanks, but the index being in-memory makes it unsuitable for large data sets :/

• iandanforth 4y ago
  Can you say more? Usually projects that gravitate to SQLlite are not those that require massive scale and a FAISS index of a few GB covers a lot of documents.
  • sharemywin 4y ago
    you might check out some vector databases: https://milvus.io/ https://milvus.io/ AND pinecone.io there are others too
  • uh_uh 4y ago
    My dataset is going to be around 10M documents. With OpenAI embeddings, that will be around 62GB. AFAIK SQLite should be able to handle that size, but I haven't tried. This is not going to be my primary DB. I would update this maybe once a day and the update doesn't have to be super fast.
• simonw 4y ago
  There is a way of running disk-backed FAISS indexed that don't all fit in memory but I've not quite figured out how to do that yet: https://github.com/facebookresearch/faiss/issues/2675 https://github.com/facebookresearch/faiss/issues/2675
  • jamesblonde 4y ago
    OpenSearch K-NN plugin supports FAISS and it's disk based: https://opensearch.org/docs/latest/search-plugins/knn/index/ https://opensearch.org/docs/latest/search-plugins/knn/index/
    • deleted 4y ago
      [deleted]
    • uh_uh 4y ago
      OpenSearch looks like the best so far, all my requirements combined!