2 ms·

Ludicrous Semantic Search Speeds

1 points by gk1 5y ago

freediver 5y ago
How about speed comparison vs a free solution like Faiss?
- gk1 5y ago
  In terms of recall and query speed, the vector search library at the core of Pinecone is on par with Faiss. But Pinecone is a distributed, managed service, like Elasticsearch on AWS, and not just a vector search library. That makes it tough to do a fair comparison with Faiss or any other vector search library for that matter. For instance: Every query from Pinecone has a slight network overhead, as described in this article. Although it's negligible in production, especially with batched queries, Faiss will always appear faster on local GPUs. And yet... People are pushing 10,000+ QPS with Pinecone because they can just set `replicas=200`, watch throughput scale linearly with each replica without affecting recall, and carry one with their day. You can do that with Faiss if you have an unhealthy obsession with Kubernetes and distributed systems (like we do :P), but then you'd just be building another Pinecone or Elasticsearch. We considered doing a comparison and then thought maybe "managed services scale better than self-hosted libraries" doesn't need explaining.
  - freediver 5y ago
    As someone obsessed with speed, I have to say that your comment missed the mark. Speed is not throughput or scale. Speed in the context of search is all about result latency. Do my users wait 400ms to get search results or 700ms? If you want to talk about ludicrous speeds, then compare it with solutions that actually provide ludicrous vector search speeds such as faiss, NGT and other libraries mentioned here http://ann-benchmarks.com http://ann-benchmarks.com and talk about what you use under the hood, innovations you made to make search faster, analyze how much network latency overhead is there when using your product and how does your (globally distributed?) infrastructure help keep that minimal. I'd be interested to read about that. What it feels like after reading your comment, is that what you are really selling is scale, not speed.
    - gk1 5y ago
      I really appreciate the feedback! We’ll tie this directly into two articles we’re already working on, one about search speed (err, latency) and another about the distributed infrastructure that makes it work.

1 points by gk1 5y ago

freediver 5y ago

How about speed comparison vs a free solution like Faiss?

gk1 5y ago
In terms of recall and query speed, the vector search library at the core of Pinecone is on par with Faiss. But Pinecone is a distributed, managed service, like Elasticsearch on AWS, and not just a vector search library. That makes it tough to do a fair comparison with Faiss or any other vector search library for that matter. For instance: Every query from Pinecone has a slight network overhead, as described in this article. Although it's negligible in production, especially with batched queries, Faiss will always appear faster on local GPUs. And yet... People are pushing 10,000+ QPS with Pinecone because they can just set `replicas=200`, watch throughput scale linearly with each replica without affecting recall, and carry one with their day. You can do that with Faiss if you have an unhealthy obsession with Kubernetes and distributed systems (like we do :P), but then you'd just be building another Pinecone or Elasticsearch. We considered doing a comparison and then thought maybe "managed services scale better than self-hosted libraries" doesn't need explaining.
- freediver 5y ago
  As someone obsessed with speed, I have to say that your comment missed the mark. Speed is not throughput or scale. Speed in the context of search is all about result latency. Do my users wait 400ms to get search results or 700ms? If you want to talk about ludicrous speeds, then compare it with solutions that actually provide ludicrous vector search speeds such as faiss, NGT and other libraries mentioned here http://ann-benchmarks.com http://ann-benchmarks.com and talk about what you use under the hood, innovations you made to make search faster, analyze how much network latency overhead is there when using your product and how does your (globally distributed?) infrastructure help keep that minimal. I'd be interested to read about that. What it feels like after reading your comment, is that what you are really selling is scale, not speed.
  - gk1 5y ago
    I really appreciate the feedback! We’ll tie this directly into two articles we’re already working on, one about search speed (err, latency) and another about the distributed infrastructure that makes it work.