Show HN: Ragas – Open-source library for evals and testing RAG systems

Ragas is an open-source library designed for evaluating and testing RAG (Retrieval-Augmented Generation) and other LLM applications. It offers a diverse set of metrics and methods, including synthetic test data generation, to help you assess your RAG applications. Ragas was initially developed to address our own needs for evaluating RAG chatbots last year. ### Problems Ragas Can Solve: - How can you select the best components for your RAG, such as the retriever, reranker, and LLM? - How can you create a test dataset without incurring significant expenses and time? We believe there's a need for an open-source standard for evaluating and testing LLM applications. Our vision is to establish this standard for the community. We're addressing this challenge by adapting ideas from the traditional ML lifecycle for LLM applications. ### ML Testing Evolved for LLM Applications Ragas is founded on the principles of metrics-driven development. Our goal is to develop and innovate techniques inspired by the latest research to address the challenges in evaluating and testing LLM applications. We don't think that merely building a sophisticated tracing tool will solve the evaluation and testing challenges. Instead, we aim to tackle these issues from a foundational level. To this end, we're introducing methods such as automated synthetic test data curation, metrics, and feedback utilization. These approaches are inspired by lessons learned from deploying stochastic models throughout our careers as machine learning engineers. While our current focus is on RAG pipelines, we intend to expand Ragas to test a broad spectrum of compound systems. This includes systems based on RAGs, agentic workflows, and various transformations. ### Try Ragas Experience Ragas by trying it out in Google Colab [here](https://colab.research.google.com/github/shahules786/openai-cookbook/blob/ragas/examples/evaluation/ragas/openai-ragas-eval-cookbook.ipynb https://colab.research.google.com/github/shahules786/openai-...). For more information, read our [documentation](https://docs.ragas.io/ https://docs.ragas.io/). We would love to hear feedback from the Hacker News community :)