6 ms·
This is pretty cool, but what are the scaling limits of a pgvector-based embeddings storage solution? 1 TB of embeddings? 100 TB? Is pgvector suitable for large
by heyitsguay 3y ago
This is pretty cool, but what are the scaling limits of a pgvector-based embeddings storage solution? 1 TB of embeddings? 100 TB? Is pgvector suitable for large scale installations?
- kiwicopple 3y agoThe scalability characteristics for HNSW haven’t been properly benchmarked yet. If it’s anything like ivfflat then it should be reasonably predictable based on memory size: https://supabase.com/blog/pgvector-performance https://supabase.com/blog/pgvector-performance I think 100TB is getting more into “sharded postgres” territory
- jkatz05 3y agoBlog author. I've done some separate testing on storing ~500GB of embeddings (~1B embeddings) in a partitioned table. The partition key was built using IVFFLAT as a "coarse quantizer" (in this case, sampling the entire dataset and finding K means), storing the mean vectors in a separate table, and then loading each vector into the partition with closest center. After that, I built an IVFFLAT index on each partition. With the indexes, this added up to ~1TB storage. This was primarily a "is it possible test" vs. thorough benchmarking.