3 ms·
Show HN: DeepEval – Evaluation and Unit Testing for LLMs
- jacky2wong 3y agoWe're thrilled to announce the launch of DeepEval, an LLM evaluation and testing suite designed to work nicely within a CI/CD pipeline. As more companies integrate LLM/RAG applications into their operations, ensuring the effectiveness, reliability and safety of these models is hard. About DeepEval We started with consulting on a few RAG projects and quickly realised how many issues came up when we iterated on our prompts, chunking methodologies, added function calls, added guardrails, etc. Very quickly we realised this had downstream effects that caused unexpected problems and results. DeepEval, inspired by Pytest, aims to make iterating on these RAG and agent applications as easy as possible by building evaluation into part of their CI/CD workflow. The goal is to make deployment of LLMs as straightforward as getting all tests to pass. Some features of DeepEval include: - Opinionated tests for answer relevancy, factual consistency, toxicness, bias. - Web UI to view tests, implementations, comparisons. - Opinionated flow for synthetic dataset creation. We are currently in a fully operational beta release and would love your feedback and suggestions to continue improving DeepEval. We are happy to answer any questions you have!
- pancakeguy 3y agoHow do I know I can trust the factual consistency metric? You linked to a research paper in the docs but reading a research paper is the last thing I want to do right now.
- jacky2wong 3y agoThanks for checking us out! We measure factual consistency using a NLP task known as "natural language inference". We then compare 2 sentences to know if one is an 'entailment' of another (for example - if "there are 3000 oranges in the supermarket", an entailment would be "there are oranges in the supermarket" - which would get a high score whereas "there are no oranges in the supermarket" would be a contradiction. We use the score of the entailment class to get a measure of how factually consistent it is. We provide a high-level summary in the documentation (https://docs.confident-ai.com/docs/measuring_llm_performance/factual_consistency https://docs.confident-ai.com/docs/measuring_llm_performance...) and will be sharing an in-depth blog article on it soon!
- alw3ys 3y agohow is this different from langsmith?
- jipster 3y agoHey, cofounder here. As far as I know, LangSmith focuses on observability (token usage, chain of thought insights, LLM cost), is vendor locked into LangChain, and uses LLMs to evaluate themselves. Here are the problems with what LangSmith offers: 1. Focusing on observability brings insights to engineering teams in terms of cost, latency, but these conclusions aren't really actionable 2. Not all teams use LangChain, plus a lot of teams who originally used LangChain to prototype is moving away as they productionize and look to improve what they've built 3. Not every LLM implementation involves a chain of thought, lLamaIndex is a good example. Apart from vendor locking, there are other use cases such as evaluating fine tuned models that are overlooked in a closed source software like LangSmith 4. Using LLMs to evaluate themselves are fine, but a lot more can be done. We offer other metrics (metrics that can be quantified from 0 - 1) such as factual consistency trained on NLI models that are much more deterministic. You can also use our package to pick and choose metrics that you care about (bias, relevancy, toxicity, etc) Hope that answers your questions.
- Superman928 3y agoThis is super helpful
- vivanpuri 3y agoInteresting repository - Wondering if there's a Guardrail integration?
- jacky2wong 3y agoWe will be adding a tutorial on how to use DeepEval as a Custom Validator for Guardrails very soon :)!