8 ms·

Absolutely! Here is a graph showing the difference. [^1] Known ID is arXiv ID that is in the vector database, Unknown IDs need the metadata to be fetched via

Absolutely! Here is a graph showing the difference. [^1] Known ID is arXiv ID that is in the vector database, Unknown IDs need the metadata to be fetched via API. Text is embedded via the model's API. FLAT and IVF_FLAT are different indexes used for the search. [^2] [1]: https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/heads/main/search_time_vs_input_type_auto_legend.png https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/he... [2]: https://zilliz.com/learn/how-to-pick-a-vector-index-in-milvus-visual-guide https://zilliz.com/learn/how-to-pick-a-vector-index-in-milvu...

binarymax 2y ago
That looks great for speed, but what about recall?
- Quizzical4230 2y ago
  That's has a major downgrade. For binary embeddings, the top 10 results are same as fp32, albeit shuffled. However after the 10th result, I think quality degrades quite a bit. I was planning to add a reranking strategy for binary embeddings. What do you think?
  - intalentive 2y ago
    Recommend reranking. You basically get full resolution performance for a negligible latency hit. (Unless you need to make two network calls…) MixedBread supports matryoshka embeddings too so that’s another option to explore on the latency-recall curve.
    - Quizzical4230 2y ago
      > Recommend reranking. Will explore it thoroughly then! > MixedBread supports matryoshka embeddings too so that’s another option to explore on the latency-recall curve. Yes, exactly why I went with this model!
  - amitness 2y ago
    Try this trick that I learned from Cohere: - Fetch top 10*k (i.e. 100) results using the hamming distance - Rerank by taking dot product between query embedding (full precision) and binary doc embeddings - Show top-10 results after re-ranking
    - Quizzical4230 2y ago
      This is pretty cool. The dot product would give the unnormalized cosine similarity from a smaller pool. Thank you so much!

binarymax 2y ago

That looks great for speed, but what about recall?

Quizzical4230 2y ago
That's has a major downgrade. For binary embeddings, the top 10 results are same as fp32, albeit shuffled. However after the 10th result, I think quality degrades quite a bit. I was planning to add a reranking strategy for binary embeddings. What do you think?
- intalentive 2y ago
  Recommend reranking. You basically get full resolution performance for a negligible latency hit. (Unless you need to make two network calls…) MixedBread supports matryoshka embeddings too so that’s another option to explore on the latency-recall curve.
  - Quizzical4230 2y ago
    > Recommend reranking. Will explore it thoroughly then! > MixedBread supports matryoshka embeddings too so that’s another option to explore on the latency-recall curve. Yes, exactly why I went with this model!
- amitness 2y ago
  Try this trick that I learned from Cohere: - Fetch top 10*k (i.e. 100) results using the hamming distance - Rerank by taking dot product between query embedding (full precision) and binary doc embeddings - Show top-10 results after re-ranking
  - Quizzical4230 2y ago
    This is pretty cool. The dot product would give the unnormalized cosine similarity from a smaller pool. Thank you so much!