You can try txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) with a Faiss backend. This Faiss wiki article might help (https://github.com/

You can try txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) with a Faiss backend. This Faiss wiki article might help (https://github.com/facebookresearch/faiss/wiki/Indexing-1G-vectors#1b-datasets https://github.com/facebookresearch/faiss/wiki/Indexing-1G-v...). For example, a partial Faiss configuration with 4-bit PQ quantization and only using 5% of the data to train an IVF index is shown below. faiss={"components": "IVF,PQ384x4fs", "sample": 0.05}