4 ms·

Show HN: Pyversity – Fast Result Diversification for Retrieval and RAG

86 points by Tananon 1y ago

Hey HN! I’ve recently open-sourced Pyversity, a lightweight library for diversifying retrieval results. Most retrieval systems optimize only for relevance, which can lead to top-k results that look almost identical. Pyversity efficiently re-ranks results to balance relevance and diversity, surfacing items that remain relevant but are less redundant. This helps with improving retrieval, recommendation, and RAG pipelines without adding latency or complexity. Main features: - Unified API: one function (diversify) supporting several well-known strategies: MMR, MSD, DPP, and COVER (with more to come) - Lightweight: the only dependency is NumPy, keeping the package small and easy to install - Fast: efficient implementations for all supported strategies; diversify results in milliseconds Re-ranking with cross-encoders is very popular right now, but also very expensive. From my experience, you can usually improve retrieval results with simpler and faster methods, such as the ones implemented in this package. This helps retrieval, recommendation, and RAG systems present richer, more informative results by ensuring each new item adds new information. Code and docs: github.com/pringled/pyversity Let me know if you have any feedback, or suggestions for other diversification strategies to support!

leobg 1y ago
Might also be useful for dataset curation, or even just prompt engineering. For example when training a classification task and picking a diverse set of examples for training or evaluation.
- Tananon 1y ago
  True, I think that's also a great usecase! Though these algorithms likely won't scale to very large datasets (e.g. millions of samples), but for smaller datasets, like fine-tuning sets, I think this would work very well. I've worked on something similar in the past that works for larger datasets (semantic deduplication: https://github.com/MinishLab/semhash https://github.com/MinishLab/semhash).
CarlosD 1y ago
Fascinating!
pu_pu 1y ago
The biggest problem with retrieval is actually semantic relevance. I think most embedding models don't really capture sentence-level semantic content and instead act more like bag-of-words models averaging local word-level information. Consider this simple test I’ve been running: Anchor: “A background service listens to a task queue and processes incoming data payloads using a custom rules engine before persisting output to a local SQLite database.” Option A (Lexical Match): “A background service listens to a message queue and processes outgoing authentication tokens using a custom hash function before transmitting output to a local SQLite database.” Option B (Semantic Match): “An asynchronous worker fetches jobs from a scheduling channel, transforms each record according to a user-defined logic system, and saves the results to an embedded relational data store on disk.” Any decent LLM (e.g., Gemini 2.5 Pro, GPT-4/5) immediately knows that the Anchor and Option B describe the same concept just with different words. But when I test embedding models like gemini-embedding-001 (currently top of MTEB), they consistently rate Option A as more similar measured by cosine similarity. They’re getting tricked by surface-level word overlap. I put together a small GitHub repo that uses ChatGPT to generate and test these “semantic triplets: https://github.com/semvec/embedstresstest https://github.com/semvec/embedstresstest gemini-embedding-001 (current #1 on MTEB leaderboard ) scored close to 0% on these adversarial examples. The repo is unpolished at the moment but it gets the idea across and everything is reproducible. Anyway, did anyone else notice this problem?
- softwaredoug 1y ago
  I’m not sure what the “biggest” problem is, but I do think diversity is vastly underappreciated compared to relevance. You can have maximally relevant search results that are horrible. Because most users (and LLMs) want to understand the range of options, not just one type of relevant option. Search for “shoes” and only see athletic shoes is a bad experience. You’ll sell more shoes, and keep the user engaged, if you show a diverse range of shoes.
  - jimmySixDOF 1y ago
    I liked how Karpathy explained part of this problem as "silent collapse" in his recent Dwarkesh podcast. Meaning the models tend to fall into a local minima situation of using a few output wording templates for a large number of similar questions, and this lack of entropy diversity it becomes a tough hard to detect problem when doing distillation or synthetic data generation in general. These algorithms as nice python functions are also useful repurposed for labeling parts of ontology and topic clusters etc [1]. Will definitely star and keep an eye on the repo ! [1] https://jina.ai/news/submodular-optimization-for-text-selection-passage-reranking-context-engineering/ https://jina.ai/news/submodular-optimization-for-text-select...
liqilin1567 1y ago
It would be better if there are some real-world performance tests and comparisons with other embedding-only search methods.
- Tananon 1y ago
  That's indeed something I plan to add in the near future. I'll probably add a tutorial as well to showcase how you can use this with e.g. sentence transformers. There's some pretty good benchmarks in the paper that I used as inspiration for some of these algorithms: https://arxiv.org/pdf/1709.05135 https://arxiv.org/pdf/1709.05135, I'll most likely try to reproduce some of these.

86 points by Tananon 1y ago

leobg 1y ago

Might also be useful for dataset curation, or even just prompt engineering. For example when training a classification task and picking a diverse set of examples for training or evaluation.

Tananon 1y ago
True, I think that's also a great usecase! Though these algorithms likely won't scale to very large datasets (e.g. millions of samples), but for smaller datasets, like fine-tuning sets, I think this would work very well. I've worked on something similar in the past that works for larger datasets (semantic deduplication: https://github.com/MinishLab/semhash https://github.com/MinishLab/semhash).

CarlosD 1y ago

Fascinating!

pu_pu 1y ago

The biggest problem with retrieval is actually semantic relevance. I think most embedding models don't really capture sentence-level semantic content and instead act more like bag-of-words models averaging local word-level information. Consider this simple test I’ve been running: Anchor: “A background service listens to a task queue and processes incoming data payloads using a custom rules engine before persisting output to a local SQLite database.” Option A (Lexical Match): “A background service listens to a message queue and processes outgoing authentication tokens using a custom hash function before transmitting output to a local SQLite database.” Option B (Semantic Match): “An asynchronous worker fetches jobs from a scheduling channel, transforms each record according to a user-defined logic system, and saves the results to an embedded relational data store on disk.” Any decent LLM (e.g., Gemini 2.5 Pro, GPT-4/5) immediately knows that the Anchor and Option B describe the same concept just with different words. But when I test embedding models like gemini-embedding-001 (currently top of MTEB), they consistently rate Option A as more similar measured by cosine similarity. They’re getting tricked by surface-level word overlap. I put together a small GitHub repo that uses ChatGPT to generate and test these “semantic triplets: https://github.com/semvec/embedstresstest https://github.com/semvec/embedstresstest gemini-embedding-001 (current #1 on MTEB leaderboard ) scored close to 0% on these adversarial examples. The repo is unpolished at the moment but it gets the idea across and everything is reproducible. Anyway, did anyone else notice this problem?

softwaredoug 1y ago
I’m not sure what the “biggest” problem is, but I do think diversity is vastly underappreciated compared to relevance. You can have maximally relevant search results that are horrible. Because most users (and LLMs) want to understand the range of options, not just one type of relevant option. Search for “shoes” and only see athletic shoes is a bad experience. You’ll sell more shoes, and keep the user engaged, if you show a diverse range of shoes.
- jimmySixDOF 1y ago
  I liked how Karpathy explained part of this problem as "silent collapse" in his recent Dwarkesh podcast. Meaning the models tend to fall into a local minima situation of using a few output wording templates for a large number of similar questions, and this lack of entropy diversity it becomes a tough hard to detect problem when doing distillation or synthetic data generation in general. These algorithms as nice python functions are also useful repurposed for labeling parts of ontology and topic clusters etc [1]. Will definitely star and keep an eye on the repo ! [1] https://jina.ai/news/submodular-optimization-for-text-selection-passage-reranking-context-engineering/ https://jina.ai/news/submodular-optimization-for-text-select...

liqilin1567 1y ago

It would be better if there are some real-world performance tests and comparisons with other embedding-only search methods.

Tananon 1y ago
That's indeed something I plan to add in the near future. I'll probably add a tutorial as well to showcase how you can use this with e.g. sentence transformers. There's some pretty good benchmarks in the paper that I used as inspiration for some of these algorithms: https://arxiv.org/pdf/1709.05135 https://arxiv.org/pdf/1709.05135, I'll most likely try to reproduce some of these.