3 ms·

I recommend readers take parent post with a grain of salt. (1) Google's offering returns with-in <5ms, in my experience. (2) the demo is for paragraphs, not sh

I recommend readers take parent post with a grain of salt. (1) Google's offering returns with-in <5ms, in my experience. (2) the demo is for paragraphs, not short text. You're putting mismatched data into the input, of course it's not going to work. Try a paragraph as suggested.

freediver 5y ago
Hmm.. There is no web service that returns response in <5ms, unless you are sitting at the very terminal of the hardware producing the output. The demo featured in this PR takes about 800-1000ms total to produce search results. How much of that is the actual API is not known. Typically, https request to an API in the cloud will cost you at least 50ms of network latency, more likely 100ms-200ms. If you are running vector search on premise you will obviously not have this overhead. Text embeddings typically work for short text as well as paragraphs (paragraph embeddings are usually mean/max of word embeddings anyway) simply because most commercial use cases demand handling of short text input (because nobody is inputting a paragraph into a typical search box; what use is a news search if you can not type a single word like 'Biden' or 'gpt3' into it).
- hiddencost 5y ago
  It's a cloud offering, so the machines are located near each other. Using it with other cloud services is a fair comparison to running it in the same box on-prem. The offering is similarity search, not a search engine. They offer image to image as another comparison point.
  - freediver 5y ago
    With the caveat of having to use GCP to host your server too, I can agree with you (although 5ms still sounds incredibly low, how many vectors was that?). I was obviously talking about a general use case where a user considers using an API like this vs running Faiss, and their server can be anywhere (a use case that is more common to me personally).
    - pierrefermat1 5y ago
      We are only talking about backend performance so I don't really think it's fair to include network latency on this metric, it can also hugely varied based on your current network/location setup
      - freediver 5y ago
        > For low-latency (up to couple of hundred milliseconds) use cases, it is highly unlikely that any cloud solution like this would be a better choice than something deployed on premise because of the *network overhead*.
- rrdharan 5y ago
  > Hmm.. There is no web service that returns response in <5ms, unless you are sitting at the very terminal of the hardware producing the output. Typical Google Cloud Bigtable P(50) is <= 4ms to a GCE VM in the same region. Source: I work on Google Cloud Bigtable. EDIT: I should clarify that applies to point reads on an established connection using gRPC-over-HTTP/2 from e.g. Java/golang/C++ client, and doesn’t include cold connection set-up / uncached auth / TLS negotiation, but those are well into P(99.999) territory if you’re doing millions of QPS (which applies to most of our large customers).
  - rrdharan 5y ago
    Also pretty sure customers using hosted Redis / Memcached offerings like Google Cloud Memorystore or AWS Elasticache etc. get sub-1ms times for service-to-VM responses. Those aren’t over HTTP of course… so perhaps they don’t count as “web services”?
    - freediver 5y ago
      I think you are talking about latency over the equipment in the same datacenter/intranet here which, yes, I do not count as web services but are more akin to 'sitting at the terminal'. If I could access Google Cloud Bigtable over the web from my home Mac at <5ms, that would of course be the biggest thing since the invention of the web :)

freediver 5y ago

Hmm.. There is no web service that returns response in <5ms, unless you are sitting at the very terminal of the hardware producing the output. The demo featured in this PR takes about 800-1000ms total to produce search results. How much of that is the actual API is not known. Typically, https request to an API in the cloud will cost you at least 50ms of network latency, more likely 100ms-200ms. If you are running vector search on premise you will obviously not have this overhead. Text embeddings typically work for short text as well as paragraphs (paragraph embeddings are usually mean/max of word embeddings anyway) simply because most commercial use cases demand handling of short text input (because nobody is inputting a paragraph into a typical search box; what use is a news search if you can not type a single word like 'Biden' or 'gpt3' into it).

hiddencost 5y ago
It's a cloud offering, so the machines are located near each other. Using it with other cloud services is a fair comparison to running it in the same box on-prem. The offering is similarity search, not a search engine. They offer image to image as another comparison point.
- freediver 5y ago
  With the caveat of having to use GCP to host your server too, I can agree with you (although 5ms still sounds incredibly low, how many vectors was that?). I was obviously talking about a general use case where a user considers using an API like this vs running Faiss, and their server can be anywhere (a use case that is more common to me personally).
  - pierrefermat1 5y ago
    We are only talking about backend performance so I don't really think it's fair to include network latency on this metric, it can also hugely varied based on your current network/location setup
    - freediver 5y ago
      > For low-latency (up to couple of hundred milliseconds) use cases, it is highly unlikely that any cloud solution like this would be a better choice than something deployed on premise because of the *network overhead*.
rrdharan 5y ago
> Hmm.. There is no web service that returns response in <5ms, unless you are sitting at the very terminal of the hardware producing the output. Typical Google Cloud Bigtable P(50) is <= 4ms to a GCE VM in the same region. Source: I work on Google Cloud Bigtable. EDIT: I should clarify that applies to point reads on an established connection using gRPC-over-HTTP/2 from e.g. Java/golang/C++ client, and doesn’t include cold connection set-up / uncached auth / TLS negotiation, but those are well into P(99.999) territory if you’re doing millions of QPS (which applies to most of our large customers).
- rrdharan 5y ago
  Also pretty sure customers using hosted Redis / Memcached offerings like Google Cloud Memorystore or AWS Elasticache etc. get sub-1ms times for service-to-VM responses. Those aren’t over HTTP of course… so perhaps they don’t count as “web services”?
  - freediver 5y ago
    I think you are talking about latency over the equipment in the same datacenter/intranet here which, yes, I do not count as web services but are more akin to 'sitting at the terminal'. If I could access Google Cloud Bigtable over the web from my home Mac at <5ms, that would of course be the biggest thing since the invention of the web :)