3 ms·
This works by streaming results from the similarity search to the filter, in descending order of similarity. As soon as enough matches are found, the similarit
by londons_explore 5y ago
This works by streaming results from the similarity search to the filter, in descending order of similarity. As soon as enough matches are found, the similarity search is terminated.
Postgres does the same thing for many queries if you look at the query plan.
Not really worthy of a blog post - especially one that says "wait till the next blog post to find out how it works!!".
- willvarfar 5y agoHow is what you describe _not_ just an efficiently implemented post filter with early out? If it turns out that’s all pinecone have, then yeah, I’m gonna be disappointed. My mind is working overtime imagining prefixing vectors with their filter terms to root everything and other naive things…
- heipei 5y agoI'd be surprised if it was anything else, given that they started with a working vector search database and then added filtering in after the fact.
- sgt101 5y agoYou have high standards about what should be in a blog post.
- gk1 5y agoNo, this isn’t at all how it works. This is not a trivial problem when dealing with vector similarity search, which Postgres does not have.
- svcrunch 5y agoVery nice writeup. I have an idea how this might be implemented. Since you rightfully want to keep the details private, let me just ask: How do you handle the case where the probed clusters contain no elements satisfying the filter criteria? Is the probe extended to additional clusters, or terminated with no results?
- WithinReason 5y agoAnd how do construct a search that can stream results in descending order of similarity, that's faster than O(N) for N values in the database?
- mnw21cam 5y agoThat's a fairly standard nearest neighbour search[0], usually made fast by using a kd-tree[1] or an R-tree[2]. [0] https://en.wikipedia.org/wiki/Nearest_neighbor_search [1] https://en.wikipedia.org/wiki/K-d_tree [2] https://en.wikipedia.org/wiki/R-tree