3 ms·

I have been looking into Postgres as a vector database option for some of the reasons you outlined in this blog. One thing I noticed when digging into pg_embedd

by zstat 3y ago

I have been looking into Postgres as a vector database option for some of the reasons you outlined in this blog. One thing I noticed when digging into pg_embedding specifically is "ef_search" controls the MAX number of neighbors you can return, which I don't think is the case for other hnsw indexes. Upping this value greatly increases the time of indexing (indexing with ef_search of 1000 took like 3-4 hours) Does the Timescale vector index limit the number of results you can return? Does that impact speed of indexing or search? I wrote a blog about our somewhat niche requirements, which includes that we need to return ~10K or more results. If curious: https://medium.com/earthrisemedia/finding-a-vector-database-to-search-the-earth-2f672cc1b5c2 https://medium.com/earthrisemedia/finding-a-vector-database-... Interested in whether that would be a possibility with Timescale.

cevian 3y ago
Hah! This was actually one of the main algorithmic challenges of adapting DiskANN to PostgreSQL. Yes, I think it's common for these algorithms to assume you know how many results to return ahead of time. But in PostgreSQL that's not how things work -- because of things like post-index-retrieval-filtering the right interface for Postgres is one that just keeps on returning more and more results until all possible matches are exhausted. We solved this by creating a "streaming" version of the search algorithm that keep state like which nodes in the graph have been visited, which have been returned etc. That's all to say -- Yes we've solved this, there are no arbitrary limits on the number of results returned.
VoVAllen 3y ago
Hi, we've solved the problem you mentioned! Please take a look on our open source postgres vector extension https://github.com/tensorchord/pgvecto.rs https://github.com/tensorchord/pgvecto.rs. Our index building process is significantly faster than pgvector on hnsw because we can utilize all the cores, whereas pgvector can only use one core. And for the filter support, we do support pre-filtering, which will guarantee enough results no matter the condition is.

by zstat 3y ago

cevian 3y ago

Hah! This was actually one of the main algorithmic challenges of adapting DiskANN to PostgreSQL. Yes, I think it's common for these algorithms to assume you know how many results to return ahead of time. But in PostgreSQL that's not how things work -- because of things like post-index-retrieval-filtering the right interface for Postgres is one that just keeps on returning more and more results until all possible matches are exhausted. We solved this by creating a "streaming" version of the search algorithm that keep state like which nodes in the graph have been visited, which have been returned etc. That's all to say -- Yes we've solved this, there are no arbitrary limits on the number of results returned.

VoVAllen 3y ago

Hi, we've solved the problem you mentioned! Please take a look on our open source postgres vector extension https://github.com/tensorchord/pgvecto.rs https://github.com/tensorchord/pgvecto.rs. Our index building process is significantly faster than pgvector on hnsw because we can utilize all the cores, whereas pgvector can only use one core. And for the filter support, we do support pre-filtering, which will guarantee enough results no matter the condition is.