6 ms·
Show HN: Ragas – Open-source library for evaluating RAG pipelines
Ragas is an open-source library for evaluating and testing RAG and other LLM applications. Github: https://docs.ragas.io/en/stable/ https://docs.ragas.io/en/stable/, docs: https://docs.ragas.io/ https://docs.ragas.io/.
Ragas provides you with different sets of metrics and methods like synthetic test data generation to help you evaluate your RAG applications. Ragas started off by scratching our own itch for evaluating our RAG chatbots last year.
Problems Ragas can solve
- How do you choose the best components for your RAG, such as the retriever, reranker, and LLM?
- How do you formulate a test dataset without spending tons of money and time?
We believe there needs to be an open-source standard for evaluating and testing LLM applications, and our vision is to build it for the community. We are tackling this challenge by evolving the ideas from the traditional ML lifecycle for LLM applications.
ML Testing Evolved for LLM Applications
We built Ragas on the principles of metrics-driven development and aim to develop and innovate techniques inspired by state-of-the-art research to solve the problems in evaluating and testing LLM applications.
We don't believe that the problem of evaluating and testing applications can be solved by building a fancy tracing tool; rather, we want to solve the problem from a layer under the stack. For this, we are introducing methods like automated synthetic test data curation, metrics, and feedback utilisation, which are inspired by lessons learned from deploying stochastic models in our careers as ML engineers.
While currently focused on RAG pipelines, our goal is to extend Ragas for testing a wide array of compound systems, including those based on RAGs, agentic workflows, and various transformations.
Try out Ragas here https://colab.research.google.com/github/shahules786/openai-cookbook/blob/ragas/examples/evaluation/ragas/openai-ragas-eval-cookbook.ipynb https://colab.research.google.com/github/shahules786/openai-... in Google Colab. Read our docs - https://docs.ragas.io/ https://docs.ragas.io/ to know more
We would love to hear feedback from the HN community :)
- dataexporter 3y agoBased on our initial analysis with RAGAS a few months ago, it didn't provide the results that our team was expecting. Required a lot of customisation on top of it. Nevertheless a pretty solid library.
- shahules 3y agoHey, thanks for trying out Ragas. As an open-source library, we are continuously improving from the feedback from the community which I see as our primary strength. I am sure that Ragas is not perfect yet, but I can assure you that it is 10x better than it was a few months ago.
- swyx 3y agocongrats on launching! i think my continuing struggle with looking at Ragas as a company/library rather than a very successful mental model is that the core of it is like 8 metrics (https://github.com/explodinggradients/ragas/tree/main/src/ragas/metrics https://github.com/explodinggradients/ragas/tree/main/src/ra...) that are each 1-200 LOC. i can inline that easily in my app and retain full control, or model that in langchain or haystack or whatever. why is Ragas a library and a company, rather than an overall "standard" or philosophy (eg like Heroku's 12 Factor Apps) that could maybe be more universally adopted without using the library? (just giving an opp to pitch some underappreciated benefits of using this library)
- shahules 3y agoThank you for asking this question. To answer this question, I will explain two directions of Ragas. The first one is the horizontal expansion of the library which involves features like - Giving you the ability to use any LLMs instantly without any hassle - Asynchronous evaluations, integrations with tracing tools, etc - Automatic support to adapt metrics to any language The second is vertical expansion or adding more core features like metrics to Ragas which includes. - Synthetic test data generation: this is something that is heavily loved by our community so we are continuously improving the quality of it. https://docs.ragas.io/en/stable/concepts/testset_generation.html https://docs.ragas.io/en/stable/concepts/testset_generation.... Now, as we expand in both directions we aim to solve the problem of how to evaluate and test compound systems. Now, to solve this we will be innovating and working on features like feedback utilization, automatically synthesizing assertions, etc to solve this hard problem. I hope I was able to answer your question. Would love to discuss more.
- swyx 3y agocool cool. so 1) will be a direct langchain competitor, and 2) is net new territory?
- shahules 3y ago1) Horizondal expansion and support are core to every framework/library. This won't make us a competitor to LC, we actually use langchain-core to support many of these like supporting different LLMs. 2) We operate in a layer underneath the stack of evals and testing because we want to solve the problem from the ground up rather than building a fancy tracing tool, which comes later in the stack.
- retrovrv 3y agoPhenomenal to see how Ragas has progressed. Congratulations on the launch
- shahules 3y agoThank you.
- jfisher4024 3y agoCongratulations on the launch! I was unable to use this library: I was trying to evaluate different non-OpenAI models and it consistently failed due to malformed JSONs coming from the model. Any thoughts about using different models? Is this just a langchain limitation?
- shahules 3y agoThanks for your feedback. We have tested Ragas on alternatives like Claude, Mixtral, Gemini, etc. Although we support all LLMs supported by Langchain, sadly many of the OSS models out of the box aren't capable of generating JSON-type output which is important for us to ensure reproducibility.
- jfisher4024 3y agoAny tips for Mixtral? That’s what we tried
- shahules 3y agoHey, I would recommend checking out our PRs. There would be PRs that have modified some of the prompts to better suit Mixtral.
- 3d27 3y agoCheckout this instead: https://github.com/confident-ai/deepeval https://github.com/confident-ai/deepeval Also has native ragas implementation but supports all models.
- nkko 3y agoCongratulations on the launch of Ragas! This looks like an incredibly valuable tool for the LLM community. As the library continues to evolve, it will be interesting to see how it adapts to handle the growing diversity of LLM architectures and use cases.
- shahules 3y agoYes, this is an interesting challenge we are also excited about.
- rhogar 3y agoCongratulations on the launch! Personally would love to see a rough estimates of the expected number of requests and tokens required to run tasks like synthetic data generation for different amounts of data. Though this is likely highly variable, would like to have a loose idea of possible incurred costs and execution time.
- shahules 3y agoHey, this is a highly requested feature. We will be implementing it soon. Something like a rough estimate is what we are planning to do.
- pawanapg 3y agoAlso check out DeepEval... our team has been using it for a while, and it's been working well for us because we can evaluate any LLMs, something this library doesn't seem to support (https://github.com/confident-ai/deepeval https://github.com/confident-ai/deepeval).
- shahules 3y agoHey, DeepEval is interesting. What do you mean by "evaluating any LLMs"?
- redskyluan 3y agoGreat product and great progress. The first step to build rage is always to evaluate. Except all the current evaluations, cost and perf should also be part of evaluations
- jjmachan 3y agocould you elaborate on what you mean by perf? cost we'll add soon
- AndrewCook71 3y agoThis is nice, we've got more Open Source LLM Evaluation Libraries coming in more often. We're using DeepEval (https://github.com/confident-ai/deepeval https://github.com/confident-ai/deepeval) currently. How is this different from that?
- shahules 3y agoDeepeval also uses Ragas underneath. They initially took a different approach by allowing uses to formulate test cases but we were focusing on RAGs only and creating metrics and features like synthetic test data generation for it. Now that we are doing good in the RAG category, we also want to expand to solve the greater challenge.