6 ms·

Vector search just got up to 10x faster and vertically scalable

92 points by gk1 4y ago

mammydady 4y ago
gk1 4y ago
For anyone interested in the code walkthrough: https://www.pinecone.io/learn/testing-p2-collections-scaling/ https://www.pinecone.io/learn/testing-p2-collections-scaling...
fdgsdfogijq 4y ago
- deleted 4y ago
  [deleted]
- MasterIdiot 4y ago
  If you really think that's enough to build a real product, go for it. Even open-source companies (Elastic, Mongo, Scylla) have to build tons of infra around their core codebase in order to make it an actual cloud product.
  - fdgsdfogijq 4y ago
    Not that easy, the founder was a director at AWS. This is just devops/obfuscation on top of an open source library: FAISS
    - gk1 4y ago
      Pinecone doesn’t use Faiss, nor ScaNN. We love Faiss and even teach people to use it[1]. There happens to be a sizable population of engineers who need more than what Faiss provides (like live index updates and metadata filtering, for example), and can’t be bothered or aren’t being paid to customize and manage open-source libraries all day. [1] https://www.pinecone.io/learn/faiss/ https://www.pinecone.io/learn/faiss/
      - fdgsdfogijq 4y ago
        So you guys developed and implemented state of the art neural network vector search from scratch? in a year? and something better than libraries with tens of contributors over years of research?
        pbadams 4y ago
        Most vector search research teams are a lot smaller than you suggest, and haven't been around that long (e.g. the FAISS paper was published in 2017). From public info, you can see they have at least one researcher working there. It's believable to me that they could have some new innovations, especially since the product space they're focusing on is different from other teams working on vector search. State-of-the-art for a specific set of constraints is still state-of-the-art. However, considering how much of their edu-marketing content is posted to HN, it would be great if they could share more details about the internals of their index with the community. One of the great things about vector search is how many techniques are open sourced or documented in papers :). Disclaimer: I work on vector search at a different company
        nl 4y ago
        Many very competitive vector search libraries are done by small teams. HNSW in NMSLIB[1] is mostly 3 people's work and it's very competitive[2]. [1] https://github.com/nmslib/nmslib https://github.com/nmslib/nmslib [2] http://ann-benchmarks.com/glove-100-angular_10_angular.html http://ann-benchmarks.com/glove-100-angular_10_angular.html
    - noogle 4y ago
      I actually built a similar solution supporting similar operations (including filtering by meta-data) using open-source libraries. Took me about 2 weeks net. I can see a clientele for such database (people who want a turnkey solution), but honestly it looks like an attempt to use a dev-ops solution to address deeper issues with problem formulation: e.g. 1. Is there really a need to search all items in the database? can subsampling make simple similarity comparison feasible? 2. Do the embeddings really need to have that many dimensions? Can we reduce their dimensionality and fit them in RAM? 3. Is embedding accurate enough compared to pairwise comparison? Can we formulate the problem to make the latter feasible? I also could not find any explanation of the underlying algorithms, especially around meta-data filtering, which is not solved by FAISS as well as their accuracy. (happy to hear otherwise)
- etaioinshrdlu 4y ago
  Which open source libraries is pinecone wrapping?
  - fdgsdfogijq 4y ago
    FAISS
    - gk1 4y ago
      This is incorrect.
      - Sommer 4y ago
        Strange comments and denial coming from the person who submitted a post about pinecone wrapping Faiss a year ago https://news.ycombinator.com/item?id=27502458 https://news.ycombinator.com/item?id=27502458
      - etaioinshrdlu 4y ago
        Yeah, what's this landing page about? https://www.pinecone.io/managed-faiss/ https://www.pinecone.io/managed-faiss/
        gk1 4y ago
        That's a very old and experimental page -- hence the "early access" mentions. Glad you pointed it out so we can delete it.
  - gk1 4y ago
    I’m not sure where the other commenter gets their confidence, but Pinecone is not wrapping any open source vector-search library. We offer three index types (in-memory, in-memory graph-based, hybrid memory + disk), and all are proprietary. We do have articles about Faiss and HNSW and all sorts of other vector-search and NLP topics, so it’s possible that’s where the confusion comes from.
    - 4y ago
phgn 4y ago
Side tangent: Pinecone pods seem to cost 15.625% more per hour on AWS compared to GCP. All the instance types hide away the price differences usually, so this is interesting to see. Edit: also there is no free tier for Pinecone on AWS :(
- deleted 4y ago
  [deleted]
opisthenar84 4y ago
I don't understand the emphasis here on vertical scaling. Move a database to a bigger machine = more storage and faster querying. Not exactly rocket science. Horizontal scaling is the real challenge here, and the complexity of vector indexes makes it especially challenging. Milvus and Vertex AI both have horizontal scaling ANN search and the ability to do parallel indexing as well. I appreciate the post but this doesn't seem worthy of an announcement.
- ww520 4y ago
  Bigger machine doesn't automatically mean higher performance. The code needs to scale with the increased number of cores, has share-nothing or share-very-little approach to avoid contention, and uses efficient data structure to utilize the increased memory.
  - marginalia_nu 4y ago
    Why would search queries have contention with each other? Surely it's in the domain of embarrassingly parallel.
    - PeterCorless 4y ago
      Disk IO needs concurrency. And you're back to Little's Law.
    - pbadams 4y ago
      I agree in the typical case, but they support concurrent add/delete one of their index options. Handling consistency/contention for modifying whatever graph/tree/etc structure they are using is probably nontrivial, and the resulting cache invalidations would also likely affect the QPS. P.S. Great work on your site, by the way - it's a really inspiring project!
      - marginalia_nu 4y ago
        Seems to me you can do that in a way that ensures low contention between consumers by using a read-biased MRSW lock. It's not free such a construction, but it really shouldn't eat into your read performance all that much. You're adding hundreds of nanoseconds to your query time by acquiring and releasing a lock. Unless you're already serving millions of queries per second per thread, this is piss in the ocean.
baobob 4y ago
Confused by their claim to be the 'first' vector database. These things have been around forever? For example FLANN (not a DB server, but example lib) is from 2009
- opisthenar84 4y ago
  I think the difference is in the layer of abstraction i.e. FLANN is just the underlying search functionality whereas vector databases are fully managed solutions. Even so, Weaviate came out in 2018, so saying that they are the "first" vector database is just flat out wrong since Pinecone was founded in 2019.
  - gk1 4y ago
    Weaviate calling themselves a vector database is a fairly new thing.
    - opisthenar84 4y ago
      The fact that Weaviate only recently started calling themselves a vector database is completely irrelevant here. They had this type of vector data infrastructure before Pinecone did, and that's all that matters. Example: I'm going to start a new company called Conifercone and do pretty much exactly what you do, but call it a "vector datastore" instead. Apparently I've now created the first ever vector datastore even though functionally I have done nothing novel.
    - thirdtrigger 4y ago
      I'm affiliated with Weaviate, so maybe nice to get this out here for the record :) We call Weaviate a "vector search engine" (i.e., we prefer "vector search engine" because it describes the type of database) since around Aug, 2020 Github: https://github.com/semi-technologies/weaviate/tree/a3967aff5ef6db380ed3b93e9879fead3ac777d6 https://github.com/semi-technologies/weaviate/tree/a3967aff5... The reason was simple; our community started to say that the mixed vector and scalar filter search capabilities were what they liked most. Also, our benchmarks are available for quite some time here: https://weaviate.io/developers/weaviate/current/benchmarks/ann.html https://weaviate.io/developers/weaviate/current/benchmarks/a... They are based on ann-benchmarks.com but adjusted for full databases.
29athrowaway 4y ago
10x faster with respect to what?
- thomas39992 4y ago
  It was more than 10x slower than Vespa, Weaviate, or Qdrant before. Maybe they are finally getting close.
learndeeply 4y ago
Didn't Milvus (vector db, wrapper around FAISS) come before Pinecone?
- fzliu 4y ago
  Just to clarify, Milvus is much more than a wrapper around FAISS. Our vector search component called Knowhere (https://github.com/milvus-io/knowhere https://github.com/milvus-io/knowhere) utilizes FAISS and Annoy and will soon include ScaNN, DiskANN, and in-house vector indexes as well. Milvus uses Knowhere as the compute engine, and implements a variety of database functions such as horizontal scaling, caching, replication, failover, and object storage on top of Knowhere. If you're interested, I recommend checking out our architecture page (https://milvus.io/docs/architecture_overview.md https://milvus.io/docs/architecture_overview.md). [EDIT]: Forgot to mention - Milvus development began in 2018 was open sourced in 2019.
johnthescott 4y ago
a custom datatype or fdw in postgresql seems interesting to me.
- MasterIdiot 4y ago
  There's one, but with some limitations (For example - only vectors of up to 1024 dimensions) https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector
kevinmm 4y ago
This is developer evangelism at its best at the behest of VCs to scale and "productionalize". There are reasons why this problem is a fundamentally difficult and coming out of the blue claiming to have found a 10x solution in a field that has attracted lots of research interest is highly sus. I would love to see a study that actually exposed their methodology, replication from independent parties and more importantly CODE.
- opisthenar84 4y ago
  If code and methodology is what you're looking for, there's some great open-source vector databases out there. Milvus: https://github.com/milvus-io/milvus https://github.com/milvus-io/milvus Qdrant: https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant Weaviate: https://github.com/semi-technologies/weaviate https://github.com/semi-technologies/weaviate Milvus seems to be the most advanced and best performing vector DB (https://www.farfetchtechblog.com/en/blog/post/powering-ai-with-vector-databases-a-benchmark-part-i/ https://www.farfetchtechblog.com/en/blog/post/powering-ai-wi...). Haven't seen Qdrant benchmarks yet but cool project nonetheless. FWIW, these open source projects are how I got into the area of vector similarity search to begin with.
wizwit999 4y ago
Its weird to see someone flex vertical scaling...

92 points by gk1 4y ago

gk1 4y ago

For anyone interested in the code walkthrough: https://www.pinecone.io/learn/testing-p2-collections-scaling/ https://www.pinecone.io/learn/testing-p2-collections-scaling...

fdgsdfogijq 4y ago

deleted 4y ago
[deleted]
MasterIdiot 4y ago
If you really think that's enough to build a real product, go for it. Even open-source companies (Elastic, Mongo, Scylla) have to build tons of infra around their core codebase in order to make it an actual cloud product.
- fdgsdfogijq 4y ago
  Not that easy, the founder was a director at AWS. This is just devops/obfuscation on top of an open source library: FAISS
  - gk1 4y ago
    Pinecone doesn’t use Faiss, nor ScaNN. We love Faiss and even teach people to use it[1]. There happens to be a sizable population of engineers who need more than what Faiss provides (like live index updates and metadata filtering, for example), and can’t be bothered or aren’t being paid to customize and manage open-source libraries all day. [1] https://www.pinecone.io/learn/faiss/ https://www.pinecone.io/learn/faiss/
    - fdgsdfogijq 4y ago
      So you guys developed and implemented state of the art neural network vector search from scratch? in a year? and something better than libraries with tens of contributors over years of research?
      - pbadams 4y ago
        Most vector search research teams are a lot smaller than you suggest, and haven't been around that long (e.g. the FAISS paper was published in 2017). From public info, you can see they have at least one researcher working there. It's believable to me that they could have some new innovations, especially since the product space they're focusing on is different from other teams working on vector search. State-of-the-art for a specific set of constraints is still state-of-the-art. However, considering how much of their edu-marketing content is posted to HN, it would be great if they could share more details about the internals of their index with the community. One of the great things about vector search is how many techniques are open sourced or documented in papers :). Disclaimer: I work on vector search at a different company
      - nl 4y ago
        Many very competitive vector search libraries are done by small teams. HNSW in NMSLIB[1] is mostly 3 people's work and it's very competitive[2]. [1] https://github.com/nmslib/nmslib https://github.com/nmslib/nmslib [2] http://ann-benchmarks.com/glove-100-angular_10_angular.html http://ann-benchmarks.com/glove-100-angular_10_angular.html
  - noogle 4y ago
    I actually built a similar solution supporting similar operations (including filtering by meta-data) using open-source libraries. Took me about 2 weeks net. I can see a clientele for such database (people who want a turnkey solution), but honestly it looks like an attempt to use a dev-ops solution to address deeper issues with problem formulation: e.g. 1. Is there really a need to search all items in the database? can subsampling make simple similarity comparison feasible? 2. Do the embeddings really need to have that many dimensions? Can we reduce their dimensionality and fit them in RAM? 3. Is embedding accurate enough compared to pairwise comparison? Can we formulate the problem to make the latter feasible? I also could not find any explanation of the underlying algorithms, especially around meta-data filtering, which is not solved by FAISS as well as their accuracy. (happy to hear otherwise)
etaioinshrdlu 4y ago
Which open source libraries is pinecone wrapping?
- fdgsdfogijq 4y ago
  FAISS
  - gk1 4y ago
    This is incorrect.
    - Sommer 4y ago
      Strange comments and denial coming from the person who submitted a post about pinecone wrapping Faiss a year ago https://news.ycombinator.com/item?id=27502458 https://news.ycombinator.com/item?id=27502458
    - etaioinshrdlu 4y ago
      Yeah, what's this landing page about? https://www.pinecone.io/managed-faiss/ https://www.pinecone.io/managed-faiss/
      - gk1 4y ago
        That's a very old and experimental page -- hence the "early access" mentions. Glad you pointed it out so we can delete it.
- gk1 4y ago
  I’m not sure where the other commenter gets their confidence, but Pinecone is not wrapping any open source vector-search library. We offer three index types (in-memory, in-memory graph-based, hybrid memory + disk), and all are proprietary. We do have articles about Faiss and HNSW and all sorts of other vector-search and NLP topics, so it’s possible that’s where the confusion comes from.
  - 4y ago

phgn 4y ago

Side tangent: Pinecone pods seem to cost 15.625% more per hour on AWS compared to GCP. All the instance types hide away the price differences usually, so this is interesting to see. Edit: also there is no free tier for Pinecone on AWS :(

deleted 4y ago
[deleted]

opisthenar84 4y ago

I don't understand the emphasis here on vertical scaling. Move a database to a bigger machine = more storage and faster querying. Not exactly rocket science. Horizontal scaling is the real challenge here, and the complexity of vector indexes makes it especially challenging. Milvus and Vertex AI both have horizontal scaling ANN search and the ability to do parallel indexing as well. I appreciate the post but this doesn't seem worthy of an announcement.

ww520 4y ago
Bigger machine doesn't automatically mean higher performance. The code needs to scale with the increased number of cores, has share-nothing or share-very-little approach to avoid contention, and uses efficient data structure to utilize the increased memory.
- marginalia_nu 4y ago
  Why would search queries have contention with each other? Surely it's in the domain of embarrassingly parallel.
  - PeterCorless 4y ago
    Disk IO needs concurrency. And you're back to Little's Law.
  - pbadams 4y ago
    I agree in the typical case, but they support concurrent add/delete one of their index options. Handling consistency/contention for modifying whatever graph/tree/etc structure they are using is probably nontrivial, and the resulting cache invalidations would also likely affect the QPS. P.S. Great work on your site, by the way - it's a really inspiring project!
    - marginalia_nu 4y ago
      Seems to me you can do that in a way that ensures low contention between consumers by using a read-biased MRSW lock. It's not free such a construction, but it really shouldn't eat into your read performance all that much. You're adding hundreds of nanoseconds to your query time by acquiring and releasing a lock. Unless you're already serving millions of queries per second per thread, this is piss in the ocean.

baobob 4y ago

Confused by their claim to be the 'first' vector database. These things have been around forever? For example FLANN (not a DB server, but example lib) is from 2009

opisthenar84 4y ago
I think the difference is in the layer of abstraction i.e. FLANN is just the underlying search functionality whereas vector databases are fully managed solutions. Even so, Weaviate came out in 2018, so saying that they are the "first" vector database is just flat out wrong since Pinecone was founded in 2019.
- gk1 4y ago
  Weaviate calling themselves a vector database is a fairly new thing.
  - opisthenar84 4y ago
    The fact that Weaviate only recently started calling themselves a vector database is completely irrelevant here. They had this type of vector data infrastructure before Pinecone did, and that's all that matters. Example: I'm going to start a new company called Conifercone and do pretty much exactly what you do, but call it a "vector datastore" instead. Apparently I've now created the first ever vector datastore even though functionally I have done nothing novel.
  - thirdtrigger 4y ago
    I'm affiliated with Weaviate, so maybe nice to get this out here for the record :) We call Weaviate a "vector search engine" (i.e., we prefer "vector search engine" because it describes the type of database) since around Aug, 2020 Github: https://github.com/semi-technologies/weaviate/tree/a3967aff5ef6db380ed3b93e9879fead3ac777d6 https://github.com/semi-technologies/weaviate/tree/a3967aff5... The reason was simple; our community started to say that the mixed vector and scalar filter search capabilities were what they liked most. Also, our benchmarks are available for quite some time here: https://weaviate.io/developers/weaviate/current/benchmarks/ann.html https://weaviate.io/developers/weaviate/current/benchmarks/a... They are based on ann-benchmarks.com but adjusted for full databases.

29athrowaway 4y ago

10x faster with respect to what?

thomas39992 4y ago
It was more than 10x slower than Vespa, Weaviate, or Qdrant before. Maybe they are finally getting close.

learndeeply 4y ago

Didn't Milvus (vector db, wrapper around FAISS) come before Pinecone?

fzliu 4y ago
Just to clarify, Milvus is much more than a wrapper around FAISS. Our vector search component called Knowhere (https://github.com/milvus-io/knowhere https://github.com/milvus-io/knowhere) utilizes FAISS and Annoy and will soon include ScaNN, DiskANN, and in-house vector indexes as well. Milvus uses Knowhere as the compute engine, and implements a variety of database functions such as horizontal scaling, caching, replication, failover, and object storage on top of Knowhere. If you're interested, I recommend checking out our architecture page (https://milvus.io/docs/architecture_overview.md https://milvus.io/docs/architecture_overview.md). [EDIT]: Forgot to mention - Milvus development began in 2018 was open sourced in 2019.

johnthescott 4y ago

a custom datatype or fdw in postgresql seems interesting to me.

MasterIdiot 4y ago
There's one, but with some limitations (For example - only vectors of up to 1024 dimensions) https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector

kevinmm 4y ago

This is developer evangelism at its best at the behest of VCs to scale and "productionalize". There are reasons why this problem is a fundamentally difficult and coming out of the blue claiming to have found a 10x solution in a field that has attracted lots of research interest is highly sus. I would love to see a study that actually exposed their methodology, replication from independent parties and more importantly CODE.

opisthenar84 4y ago
If code and methodology is what you're looking for, there's some great open-source vector databases out there. Milvus: https://github.com/milvus-io/milvus https://github.com/milvus-io/milvus Qdrant: https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant Weaviate: https://github.com/semi-technologies/weaviate https://github.com/semi-technologies/weaviate Milvus seems to be the most advanced and best performing vector DB (https://www.farfetchtechblog.com/en/blog/post/powering-ai-with-vector-databases-a-benchmark-part-i/ https://www.farfetchtechblog.com/en/blog/post/powering-ai-wi...). Haven't seen Qdrant benchmarks yet but cool project nonetheless. FWIW, these open source projects are how I got into the area of vector similarity search to begin with.

wizwit999 4y ago

Its weird to see someone flex vertical scaling...