4 ms·

Thanks for sharing. How does this compare with DiskANN (https://zilliz.com/blog/diskann-a-disk-based-anns-solution-with-high-recall-and-high-qps-on-billion-scal

by fzliu 3y ago

Thanks for sharing. How does this compare with DiskANN (https://zilliz.com/blog/diskann-a-disk-based-anns-solution-with-high-recall-and-high-qps-on-billion-scale-dataset https://zilliz.com/blog/diskann-a-disk-based-anns-solution-w...) or HNSW-IF (https://blog.vespa.ai/vespa-hybrid-billion-scale-vector-search/ https://blog.vespa.ai/vespa-hybrid-billion-scale-vector-sear...)?

roseway4 3y ago
This is comparing apples and oranges. You've pointed to some technologies from vector database vendors. Neon isn't a vector database vendor. They're a Postgres vendor. The objective of pg_embedding, pg_vector, and others is to offer teams who deploy Postgres the opportunity to use their existing infrastructure for vector search. The added and important benefit here is hybrid search: using existing DB data to filter semantic search results. What the work done by Neon, the pgvector team, Supabase and others points to is that "speed" isn't the only factor in vector database selection. Developer experience and existing infrastructure investment are too.
- HammadB 3y ago
  No one is discrediting (or even discussing really) the validity of the point that if you use postgres pgvector or pgembedding may makes sense for the reasons you mention. The question was about how the algorithms compare, I compared the algorithms. Comparing ANN algorithms is relatively apples-to-apples, but is still complicated and filled with nuance.
HammadB 3y ago
DiskANN construction is much much slower than HNSW, and the support around updates has been discussed in literature but has not been open sourced yet (They call this FreshDiskANN in the paper - DiskANN is from microsoft research btw). HNSW-IF is an excellent extension to HNSW (that the vespa team has made easy to implement) that takes advantage of the speed/recall of HNSW in combination with the disk scalability of inverted indices - it is a hybrid approach.
- mattashii 3y ago
  I've done just a little reading on ANN indexes and data structures, but considering you're at Chroma, could you explain why closeness (inverted distance) would be used in HNSW-IF instead of distance?
nikita 3y ago
DiskANN has some great ideas that will likely make it into pg_embedding. The most important implication is how many vectors it can support. DiskANN claims it can manage 1Bln vectors and it's a huge deal. Index construction does take a long time, but we are researching how we can scale it out.

by fzliu 3y ago

roseway4 3y ago

This is comparing apples and oranges. You've pointed to some technologies from vector database vendors. Neon isn't a vector database vendor. They're a Postgres vendor. The objective of pg_embedding, pg_vector, and others is to offer teams who deploy Postgres the opportunity to use their existing infrastructure for vector search. The added and important benefit here is hybrid search: using existing DB data to filter semantic search results. What the work done by Neon, the pgvector team, Supabase and others points to is that "speed" isn't the only factor in vector database selection. Developer experience and existing infrastructure investment are too.

HammadB 3y ago
No one is discrediting (or even discussing really) the validity of the point that if you use postgres pgvector or pgembedding may makes sense for the reasons you mention. The question was about how the algorithms compare, I compared the algorithms. Comparing ANN algorithms is relatively apples-to-apples, but is still complicated and filled with nuance.

HammadB 3y ago

DiskANN construction is much much slower than HNSW, and the support around updates has been discussed in literature but has not been open sourced yet (They call this FreshDiskANN in the paper - DiskANN is from microsoft research btw). HNSW-IF is an excellent extension to HNSW (that the vespa team has made easy to implement) that takes advantage of the speed/recall of HNSW in combination with the disk scalability of inverted indices - it is a hybrid approach.

mattashii 3y ago
I've done just a little reading on ANN indexes and data structures, but considering you're at Chroma, could you explain why closeness (inverted distance) would be used in HNSW-IF instead of distance?

nikita 3y ago

DiskANN has some great ideas that will likely make it into pg_embedding. The most important implication is how many vectors it can support. DiskANN claims it can manage 1Bln vectors and it's a huge deal. Index construction does take a long time, but we are researching how we can scale it out.