3 ms·
Can you say more? Usually projects that gravitate to SQLlite are not those that require massive scale and a FAISS index of a few GB covers a lot of documents.
by iandanforth 4y ago
Can you say more? Usually projects that gravitate to SQLlite are not those that require massive scale and a FAISS index of a few GB covers a lot of documents.
- sharemywin 4y agoyou might check out some vector databases: https://milvus.io/ https://milvus.io/ AND pinecone.io there are others too
- uh_uh 4y agoMy dataset is going to be around 10M documents. With OpenAI embeddings, that will be around 62GB. AFAIK SQLite should be able to handle that size, but I haven't tried. This is not going to be my primary DB. I would update this maybe once a day and the update doesn't have to be super fast.