6 ms·

A gentle introduction to vector databases

133 points by fzliu 5y ago

phenkdo 5y ago
Nice writeup. Have you looked at qdrant [1] for your comparison? I found it better than Milvus. [1] https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant
- occupant 5y ago
  What did you find better about it?
  - phenkdo 5y ago
    For us the winning feature for qdrant it has the concept of both positive and negative query vectors. Overall they are both very good! It's a matter of individual preference.
joexner 5y ago
Vector indices are the novel part of vector databases. Let's hear more about them. The rest is just BLOB CRUD.
- dontreact 5y ago
  The way that vector indices work typically can make doing CRUD with them a real challenge. There is definitely novelty in being able to do both ANN indexing and fast high throughput CRUD. In addition, the R of crud is hard to combine with vector indices. Case in point I am still waiting for elastic search to support both ANN and regular, structured filtering together well.
  - lmeyerov 5y ago
    I saw elastic has vector indexes now -- what does 'well' mean? My guess is for most popular uses, vector indexes are indeed more of a feature than a product, so will smooth out over time in big DBs, and curious what that should look like. Seperately, I do think there is a small niche for opinionated vector-first DBs, and not sure what that is either :)
    - anentropic 5y ago
      > I saw elastic has vector indexes now -- what does 'well' mean? Funny enough I was just reading about this today as a result of the OP article A commenter here, who apparently works for the pinecone.io vector db platform, states that: "Pre-filtering in most solutions (eg, Elasticsearch on AWS) requires using an inefficient brute-force search (kNN) through the remaining vectors after they've been filtered, because the original index was built on the unfiltered list and would no longer be useful. This causes sky-high search latencies." https://www.reddit.com/r/MachineLearning/comments/po3pos/comment/hcy4e4w/?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/MachineLearning/comments/po3pos/com...
      - zmjjmz 5y ago
        We've had good success with Vespa [1] for this usecase, both using it to rank by vector similarity and apply business logic oriented filters. Search latencies are fairly low, even with constantly updating documents. [1] https://tech.okcupid.com/vespa-vs-elasticsearch-for-matching-millions-of-people-6e3af18eb4dc https://tech.okcupid.com/vespa-vs-elasticsearch-for-matching...
        mrintellectual 5y ago
        I've heard great things about Vespa as well (I used to work at Yahoo). I unfortunately never got the chance to play around with it too much myself. Looking forward to doing some at some point in the near future though.
      - gk1 5y ago
        That was me. In all honesty if you are already using ES and you just want nearest-neighbor search for less than 10M documents, just stay with ES. Things get less obvious when you grow past 10M documents and still want low latency. Or if you need live index updates without downtime, or if you want to apply metadata filters to nearest-neighbor searches. If you have 100M documents -- not a difficult threshold if you're an enterprise software company or a popular consumer app -- then ES gets ruled out fairly early in the process. We get a lot of those exasperated teams coming to Pinecone after trying their best with ES/OpenSearch.
  - gk1 5y ago
    Pinecone does this, FYI: https://www.pinecone.io/docs/metadata-filtering/ https://www.pinecone.io/docs/metadata-filtering/ And you're not alone... Lots of people want this, and lots of people can't get it with ES.
    - dontreact 5y ago
      Yes but my sense it is coming pretty soon for ES, and switching to Pinecone has a huge rewrite cost. As NNs keep getting better, vector search keeps becoming more important, and it becomes more and more existential for elasticsearch to do this well. I wonder if there actually some fundamental reason they won't be able to do so.
      - gk1 5y ago
        We've had people kicking down our doors (figuratively) for the filtering feature. I have no doubt ES is getting the same feedback and is working on it. > I wonder if there actually some fundamental reason they won't be able to do so. This is where it gets interesting. I don't think there's anything a company with ES's resources can't do if they really want to. However... ES's vector search is based on Lucene 9.0, which in turn uses the HNSW vector index. HNSW is shaky when it comes to CRUD updates (namely deletions), and flat out does not support metadata filtering: https://issues.apache.org/jira/browse/LUCENE-10040 https://issues.apache.org/jira/browse/LUCENE-10040 Even with ES resources, it's a fundamentally hard problem. In the best case it's going to take a while to implement.
        peterstjohn 5y ago
        If you control the HNSW implementation, it can definitely do pre-filtering. Vespa does it, and you can modify open source HNSW libs easily. I added pre-filtering support to an internal fork of HNSWLIB last week, for example…
  - heipei 5y ago
    If anyone wants to build a scalable ANN-index with single-stage filtering (i.e. not with the builtin vector index which does post-filtering), I suggest people try binarising and splitting their feature vectors, or using something like Product Quantization (PQ). Both approaches will return a list of fixed terms which can be indexed in Elasticsearch as keyword and then searched with a simple "terms" query. Big fan of what Pinecone is doing, but I have too much invested into Elasticsearch/Lucene at this point in time to be considering anything else really, and with Elasticsearch I get everything in one box, including things like n-gram accelerated wildcard searches.
- gk1 5y ago
  Here you go: https://www.pinecone.io/learn/vector-indexes/ https://www.pinecone.io/learn/vector-indexes/
  - deleted 5y ago
    [deleted]
  - xibalba 5y ago
    Hi, could you please stop marketing here? You've posted a bunch of replies in this thread linking to your employer. It's nice that you're (sort of) disclosing your biases, but we really don't want HN to turn into marketing spam.
    - gk1 5y ago
      Yep, Dang already pointed out how my comments might come across as spammy, so I acknowledge that and apologize! No spam intended, just hoping to be helpful. I'll tone things down.
- mrintellectual 5y ago
  Thanks for your feedback. I'm writing a post on vector indices and will throw it up this week.
splatcollision 5y ago
I've done vector indexes on CouchDB before, because it supports arrays as keys in it's map-reduce implementation. Worked great for fast document similarity search. Brief writeup: http://splatcollision.com/page/fast-vector-similarity-queries-using-couchdb-views http://splatcollision.com/page/fast-vector-similarity-querie...
gk1 5y ago
This is a great writeup, and awesome to see vector databases come up more and more often. For anyone interested in going down this rabbit hole, we have an entire learning center about vector databases and vector search (https://www.pinecone.io/learn/ https://www.pinecone.io/learn/) including the obligatory "What is a Vector Database" intro with example notebooks: https://www.pinecone.io/learn/vector-database/ https://www.pinecone.io/learn/vector-database/
- starkd 5y ago
  Thank you for this. One approach I find missing in your blog is that of distance-based indexing. It's an approach that indexes vectors according to distances from chosen vantage points from within the data set. I've done some preliminary work on creating a system for images: phash.dev
- liminal 5y ago
  Pinecone looks great. Any plans to have a non-hosted option?
- dang 5y ago
  You've posted several comments in this thread alone linking to your product, and it seems that the majority of your posts have been doing this for quite a while now. I'm sure it's excellent work, but can you please stop doing this? It's fine to link to your own work occasionally, when it's particularly relevant, as part of a diverse mix of posts on unrelated things*. It's not ok to use HN primarily for promotion. See https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html: "Please don't use HN primarily for promotion. It's ok to post your own stuff occasionally, but the primary use of the site should be for curiosity." When people do that we eventually start penalizing their accounts and sites, or in egregious cases, banning them. You're a good HN user, but this is still excessive. You're crossing the line at which the community starts to think of the word 'spam', and we inevitably start getting emails about it. * I do get that your work is particularly relevant in a thread like this. What's missing is the 'diverse mix of posts on unrelated things'. In such a context, posting repeatedly about your own stuff starts to come across the wrong way.
mrintellectual 5y ago
As mentioned in the article, I recommend Milvus (https://milvus.io https://milvus.io) - it's open source and cloud native with standalone versions available. Alternatively, if you're looking for an open-source solution for generating embeddings, I recommend (https://github.com/towhee-io/towhee https://github.com/towhee-io/towhee).
- dtjohnnyb 5y ago
  One downside for milvus is that version 1 doesn't do filtering (necessary for most search applications) and version 2 is significantly slower. Google's vector nearest neighbors offering, weaviate, and Vespa are much better options if you're expecting to extend to more realistic workloads
tabtab 5y ago
This reminds me of "Factor Tables": https://github.com/RowColz/AI https://github.com/RowColz/AI
cbsmith 5y ago
Everything old is new again. ;-)
krishnakatyal 5y ago
Very well written

133 points by fzliu 5y ago

Nice writeup. Have you looked at qdrant [1] for your comparison? I found it better than Milvus. [1] https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant

occupant 5y ago
What did you find better about it?
- phenkdo 5y ago
  For us the winning feature for qdrant it has the concept of both positive and negative query vectors. Overall they are both very good! It's a matter of individual preference.

joexner 5y ago

Vector indices are the novel part of vector databases. Let's hear more about them. The rest is just BLOB CRUD.

dontreact 5y ago
The way that vector indices work typically can make doing CRUD with them a real challenge. There is definitely novelty in being able to do both ANN indexing and fast high throughput CRUD. In addition, the R of crud is hard to combine with vector indices. Case in point I am still waiting for elastic search to support both ANN and regular, structured filtering together well.
- lmeyerov 5y ago
  I saw elastic has vector indexes now -- what does 'well' mean? My guess is for most popular uses, vector indexes are indeed more of a feature than a product, so will smooth out over time in big DBs, and curious what that should look like. Seperately, I do think there is a small niche for opinionated vector-first DBs, and not sure what that is either :)
  - anentropic 5y ago
    > I saw elastic has vector indexes now -- what does 'well' mean? Funny enough I was just reading about this today as a result of the OP article A commenter here, who apparently works for the pinecone.io vector db platform, states that: "Pre-filtering in most solutions (eg, Elasticsearch on AWS) requires using an inefficient brute-force search (kNN) through the remaining vectors after they've been filtered, because the original index was built on the unfiltered list and would no longer be useful. This causes sky-high search latencies." https://www.reddit.com/r/MachineLearning/comments/po3pos/comment/hcy4e4w/?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/MachineLearning/comments/po3pos/com...
    - zmjjmz 5y ago
      We've had good success with Vespa [1] for this usecase, both using it to rank by vector similarity and apply business logic oriented filters. Search latencies are fairly low, even with constantly updating documents. [1] https://tech.okcupid.com/vespa-vs-elasticsearch-for-matching-millions-of-people-6e3af18eb4dc https://tech.okcupid.com/vespa-vs-elasticsearch-for-matching...
      - mrintellectual 5y ago
        I've heard great things about Vespa as well (I used to work at Yahoo). I unfortunately never got the chance to play around with it too much myself. Looking forward to doing some at some point in the near future though.
    - gk1 5y ago
      That was me. In all honesty if you are already using ES and you just want nearest-neighbor search for less than 10M documents, just stay with ES. Things get less obvious when you grow past 10M documents and still want low latency. Or if you need live index updates without downtime, or if you want to apply metadata filters to nearest-neighbor searches. If you have 100M documents -- not a difficult threshold if you're an enterprise software company or a popular consumer app -- then ES gets ruled out fairly early in the process. We get a lot of those exasperated teams coming to Pinecone after trying their best with ES/OpenSearch.
- gk1 5y ago
  Pinecone does this, FYI: https://www.pinecone.io/docs/metadata-filtering/ https://www.pinecone.io/docs/metadata-filtering/ And you're not alone... Lots of people want this, and lots of people can't get it with ES.
  - dontreact 5y ago
    Yes but my sense it is coming pretty soon for ES, and switching to Pinecone has a huge rewrite cost. As NNs keep getting better, vector search keeps becoming more important, and it becomes more and more existential for elasticsearch to do this well. I wonder if there actually some fundamental reason they won't be able to do so.
    - gk1 5y ago
      We've had people kicking down our doors (figuratively) for the filtering feature. I have no doubt ES is getting the same feedback and is working on it. > I wonder if there actually some fundamental reason they won't be able to do so. This is where it gets interesting. I don't think there's anything a company with ES's resources can't do if they really want to. However... ES's vector search is based on Lucene 9.0, which in turn uses the HNSW vector index. HNSW is shaky when it comes to CRUD updates (namely deletions), and flat out does not support metadata filtering: https://issues.apache.org/jira/browse/LUCENE-10040 https://issues.apache.org/jira/browse/LUCENE-10040 Even with ES resources, it's a fundamentally hard problem. In the best case it's going to take a while to implement.
      - peterstjohn 5y ago
        If you control the HNSW implementation, it can definitely do pre-filtering. Vespa does it, and you can modify open source HNSW libs easily. I added pre-filtering support to an internal fork of HNSWLIB last week, for example…
- heipei 5y ago
  If anyone wants to build a scalable ANN-index with single-stage filtering (i.e. not with the builtin vector index which does post-filtering), I suggest people try binarising and splitting their feature vectors, or using something like Product Quantization (PQ). Both approaches will return a list of fixed terms which can be indexed in Elasticsearch as keyword and then searched with a simple "terms" query. Big fan of what Pinecone is doing, but I have too much invested into Elasticsearch/Lucene at this point in time to be considering anything else really, and with Elasticsearch I get everything in one box, including things like n-gram accelerated wildcard searches.
gk1 5y ago
Here you go: https://www.pinecone.io/learn/vector-indexes/ https://www.pinecone.io/learn/vector-indexes/
- deleted 5y ago
  [deleted]
- xibalba 5y ago
  Hi, could you please stop marketing here? You've posted a bunch of replies in this thread linking to your employer. It's nice that you're (sort of) disclosing your biases, but we really don't want HN to turn into marketing spam.
  - gk1 5y ago
    Yep, Dang already pointed out how my comments might come across as spammy, so I acknowledge that and apologize! No spam intended, just hoping to be helpful. I'll tone things down.
mrintellectual 5y ago
Thanks for your feedback. I'm writing a post on vector indices and will throw it up this week.

splatcollision 5y ago

I've done vector indexes on CouchDB before, because it supports arrays as keys in it's map-reduce implementation. Worked great for fast document similarity search. Brief writeup: http://splatcollision.com/page/fast-vector-similarity-queries-using-couchdb-views http://splatcollision.com/page/fast-vector-similarity-querie...

gk1 5y ago

This is a great writeup, and awesome to see vector databases come up more and more often. For anyone interested in going down this rabbit hole, we have an entire learning center about vector databases and vector search (https://www.pinecone.io/learn/ https://www.pinecone.io/learn/) including the obligatory "What is a Vector Database" intro with example notebooks: https://www.pinecone.io/learn/vector-database/ https://www.pinecone.io/learn/vector-database/

starkd 5y ago
Thank you for this. One approach I find missing in your blog is that of distance-based indexing. It's an approach that indexes vectors according to distances from chosen vantage points from within the data set. I've done some preliminary work on creating a system for images: phash.dev
liminal 5y ago
Pinecone looks great. Any plans to have a non-hosted option?
dang 5y ago
You've posted several comments in this thread alone linking to your product, and it seems that the majority of your posts have been doing this for quite a while now. I'm sure it's excellent work, but can you please stop doing this? It's fine to link to your own work occasionally, when it's particularly relevant, as part of a diverse mix of posts on unrelated things*. It's not ok to use HN primarily for promotion. See https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html: "Please don't use HN primarily for promotion. It's ok to post your own stuff occasionally, but the primary use of the site should be for curiosity." When people do that we eventually start penalizing their accounts and sites, or in egregious cases, banning them. You're a good HN user, but this is still excessive. You're crossing the line at which the community starts to think of the word 'spam', and we inevitably start getting emails about it. * I do get that your work is particularly relevant in a thread like this. What's missing is the 'diverse mix of posts on unrelated things'. In such a context, posting repeatedly about your own stuff starts to come across the wrong way.

mrintellectual 5y ago

As mentioned in the article, I recommend Milvus (https://milvus.io https://milvus.io) - it's open source and cloud native with standalone versions available. Alternatively, if you're looking for an open-source solution for generating embeddings, I recommend (https://github.com/towhee-io/towhee https://github.com/towhee-io/towhee).

dtjohnnyb 5y ago
One downside for milvus is that version 1 doesn't do filtering (necessary for most search applications) and version 2 is significantly slower. Google's vector nearest neighbors offering, weaviate, and Vespa are much better options if you're expecting to extend to more realistic workloads

tabtab 5y ago

This reminds me of "Factor Tables": https://github.com/RowColz/AI https://github.com/RowColz/AI

cbsmith 5y ago

Everything old is new again. ;-)

krishnakatyal 5y ago

Very well written