You can do pretty fast (single digit ms) cosine similarity on 1m vectors with numpy on the CPU. And there are small embedded databases like RocksDB with HNSW in

You can do pretty fast (single digit ms) cosine similarity on 1m vectors with numpy on the CPU. And there are small embedded databases like RocksDB with HNSW indices that work well beyond that. Especially staying at like 256 dims. Quantization can shrink even further. Also locally you’re QPS is very low as you’re the only searcher. So with enough RAM, and small enough dataset, it should be fine.