5 ms·

The embeddings/search API seem super powerful, been meaning to play around with it more. I wonder how its performance compares to ElasticSearch / other text sea

The embeddings/search API seem super powerful, been meaning to play around with it more. I wonder how its performance compares to ElasticSearch / other text search/classification offerings out there

gk1 4y ago
The Search and Classification (and Answers) APIs were deprecated last week.[1] They were never in serious competition with Elastic, as far as search goes. If you wanted to build a semantic search application using OpenAI embeddings, the more common (and scalable) method is to index those embeddings in a vector database like Pinecone.[2] In fact that's what OpenAI recommends to anyone who needs to transition off their Search API. [1] https://help.openai.com/en/articles/6272952-search-transition-guide https://help.openai.com/en/articles/6272952-search-transitio... [2] https://docs.pinecone.io/docs/openai https://docs.pinecone.io/docs/openai
- thirdtrigger 4y ago
  Agreed – one can also use Weaviate which comes with an OOTB OpenAI module leveraging the embeddings end-point https://weaviate.io/developers/weaviate/current/retriever-vectorizer-modules/text2vec-openai.html https://weaviate.io/developers/weaviate/current/retriever-ve...
- lmeyerov 4y ago
  I'm curious why you say 'more common'. When we did some adoption analysis, we found elastic vector indexes to have much more traction than others (excluding say faiss), even if it's a small % of elastic's advertising. How did you come to finding pinecone is the more common method?
  - gk1 4y ago
    I meant OpenAI + Vector Database as more common than OpenAI Search API. And then Pinecone as an example of a vector database.
    - lmeyerov 4y ago
      Ah yes, thanks! Yeah we saw faiss + es leaders for serving embeddings / vector search, and pinecone / weaviate / I think milvus as next tier, so was curious if we could improve the analysis :)
      - gk1 4y ago
        Is the analysis public? I'm curious how you determined the popularity of a product like Pinecone, since we don't have a public metric like GitHub stars.
        lmeyerov 4y ago
        Loosely measure search, CVs, and job ads: https://gradientflow.com/the-vector-database-index/ https://gradientflow.com/the-vector-database-index/ This kind of analysis is rarely precise, but is useful for rougher tasks like tiering My personal question is if vector indexes are/will be a good-enough general DB feature / compute lib for most users & use cases. A lucrative niche market can still happen as VC dollars disappear, similar to graph DBs, so not a knock, just important for folks deciding how to build things.
      - andre-z 4y ago
        FAISS is depreciating since it is just a library that does not scale and lacks a lot of vector db functionalities like filtering. ES is often used just by default because Devs have experience with but the performance is very low compared to dedicated solutions. The Github trending for Vector Databases https://github.com/topics/vector-database https://github.com/topics/vector-database
CaptainNegative 4y ago
Elastic/OpenSearch have classically used BM25, but the latter recently added semantic search capabilities using neural embeddings in 2.4. Not sure about the former. [1] https://opensearch.org/blog/opensearch-2-4-is-available-today/ https://opensearch.org/blog/opensearch-2-4-is-available-toda...
- andre-z 4y ago
  ES and OS are desperately slow because based on the lucene vector search index. A dedicated vector database like Qdrant will be always a better choice https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant
  - reschkek 4y ago
    Do we have any idea why lucene vector search underperforms? As of lucene 9.1 (and elastic 8.4), it runs the same sort of filtered/categorical HNSW that qdrant runs (https://lucene.apache.org/core/9_1_0/core/org/apache/lucene/search/KnnVectorQuery.html https://lucene.apache.org/core/9_1_0/core/org/apache/lucene/...). Qdrant's benchmarking code (https://github.com/qdrant/vector-db-benchmark/blob/9263ba/engine/clients/elasticsearch/search.py https://github.com/qdrant/vector-db-benchmark/blob/9263ba/en...) does use the new filtered ann query with elastic 8.4, so it appears to be a fair benchmark. Why is lucene/elastic so much slower? Is it a rust vs. java thing? Or some memory management issues?

The embeddings/search API seem super powerful, been meaning to play around with it more. I wonder how its performance compares to ElasticSearch / other text search/classification offerings out there

gk1 4y ago

The Search and Classification (and Answers) APIs were deprecated last week.[1] They were never in serious competition with Elastic, as far as search goes. If you wanted to build a semantic search application using OpenAI embeddings, the more common (and scalable) method is to index those embeddings in a vector database like Pinecone.[2] In fact that's what OpenAI recommends to anyone who needs to transition off their Search API. [1] https://help.openai.com/en/articles/6272952-search-transition-guide https://help.openai.com/en/articles/6272952-search-transitio... [2] https://docs.pinecone.io/docs/openai https://docs.pinecone.io/docs/openai

thirdtrigger 4y ago
Agreed – one can also use Weaviate which comes with an OOTB OpenAI module leveraging the embeddings end-point https://weaviate.io/developers/weaviate/current/retriever-vectorizer-modules/text2vec-openai.html https://weaviate.io/developers/weaviate/current/retriever-ve...
lmeyerov 4y ago
I'm curious why you say 'more common'. When we did some adoption analysis, we found elastic vector indexes to have much more traction than others (excluding say faiss), even if it's a small % of elastic's advertising. How did you come to finding pinecone is the more common method?
- gk1 4y ago
  I meant OpenAI + Vector Database as more common than OpenAI Search API. And then Pinecone as an example of a vector database.
  - lmeyerov 4y ago
    Ah yes, thanks! Yeah we saw faiss + es leaders for serving embeddings / vector search, and pinecone / weaviate / I think milvus as next tier, so was curious if we could improve the analysis :)
    - gk1 4y ago
      Is the analysis public? I'm curious how you determined the popularity of a product like Pinecone, since we don't have a public metric like GitHub stars.
      - lmeyerov 4y ago
        Loosely measure search, CVs, and job ads: https://gradientflow.com/the-vector-database-index/ https://gradientflow.com/the-vector-database-index/ This kind of analysis is rarely precise, but is useful for rougher tasks like tiering My personal question is if vector indexes are/will be a good-enough general DB feature / compute lib for most users & use cases. A lucrative niche market can still happen as VC dollars disappear, similar to graph DBs, so not a knock, just important for folks deciding how to build things.
    - andre-z 4y ago
      FAISS is depreciating since it is just a library that does not scale and lacks a lot of vector db functionalities like filtering. ES is often used just by default because Devs have experience with but the performance is very low compared to dedicated solutions. The Github trending for Vector Databases https://github.com/topics/vector-database https://github.com/topics/vector-database

CaptainNegative 4y ago

Elastic/OpenSearch have classically used BM25, but the latter recently added semantic search capabilities using neural embeddings in 2.4. Not sure about the former. [1] https://opensearch.org/blog/opensearch-2-4-is-available-today/ https://opensearch.org/blog/opensearch-2-4-is-available-toda...

andre-z 4y ago
ES and OS are desperately slow because based on the lucene vector search index. A dedicated vector database like Qdrant will be always a better choice https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant
- reschkek 4y ago
  Do we have any idea why lucene vector search underperforms? As of lucene 9.1 (and elastic 8.4), it runs the same sort of filtered/categorical HNSW that qdrant runs (https://lucene.apache.org/core/9_1_0/core/org/apache/lucene/search/KnnVectorQuery.html https://lucene.apache.org/core/9_1_0/core/org/apache/lucene/...). Qdrant's benchmarking code (https://github.com/qdrant/vector-db-benchmark/blob/9263ba/engine/clients/elasticsearch/search.py https://github.com/qdrant/vector-db-benchmark/blob/9263ba/en...) does use the new filtered ann query with elastic 8.4, so it appears to be a fair benchmark. Why is lucene/elastic so much slower? Is it a rust vs. java thing? Or some memory management issues?