jacky2wong

16 ms·

1. 61.▲
   PDB Support for DeepEval(docs.confident-ai.com)

   1 points by jacky2wong 3y ago|0 comments
2. 62.▲
   Test for Bias After Finetuning LLMs(docs.confident-ai.com)

   1 points by jacky2wong 3y ago|0 comments
3. 63.▲
   Measure Answer Relevancy of LLMs(docs.confident-ai.com)

   1 points by jacky2wong 3y ago|0 comments
4. 64.▲
   Test LLMs for Toxicness(docs.confident-ai.com)

   1 points by jacky2wong 3y ago|0 comments
5. 65.▲
   Auto-Evaluation of LLMs with DeepEval(docs.confident-ai.com)

   2 points by jacky2wong 3y ago|1 comments
6. 66.▲
   by jacky2wong 3y ago

   One complaint we heard over and over was that no one likes to write tests when evaluating LLMs. We automate writing these tests by having ChatGPT generate sample queries and answers based on a given text. These test cases can then be run th
7. 67.▲
   Test for LLM Hallucinations(docs.confident-ai.com)

   2 points by jacky2wong 3y ago|0 comments
8. 68.▲
   DeepEval – Neural Framework for Testing LLMs(github.com)

   2 points by jacky2wong 3y ago|0 comments
9. 69.▲
   by jacky2wong 3y ago

   For those that need help migrating their models to OpenAI enterprise and ensure consistency results - would recommend checking the open source repo https://github.com/confident-ai/deepeval
10. 70.▲
    by jacky2wong 3y ago

    To be honest - I had the opposite feeling - I found it pretty easy to wrap my head around their abstractions. If anything - I believe their abstractions are quite clean as I was able to navigate their codebase quite easily. Their "cont
11. 71.▲
    Unit Test LlamaIndex with DeepEval(docs.confident-ai.com)

    35 points by jacky2wong 3y ago|3 comments
12. 72.▲
    Unit Testing for Rag(github.com)

    2 points by jacky2wong 3y ago|0 comments
13. 73.▲
    DeepEval CLI(github.com)

    2 points by jacky2wong 3y ago|0 comments
14. 74.▲
    Testing Rank Similarity for Rag(docs.confident-ai.com)

    1 points by jacky2wong 3y ago|0 comments
15. 75.▲
    Framework for evaluating LLM outputs with ML models(docs.confident-ai.com)

    2 points by jacky2wong 3y ago|0 comments
16. 76.▲
    Mental Model of Prisma Migrate(prisma.io)

    1 points by jacky2wong 3y ago|0 comments
17. 77.▲
    How to test LLM is non-toxic before pushing to prod(docs.confident-ai.com)

    2 points by jacky2wong 3y ago|0 comments
18. 78.▲
    Testing for Factual Consistency in LLMs(docs.confident-ai.com)

    1 points by jacky2wong 3y ago|0 comments
19. 79.▲
    How to measure ranking similarity for RAG systems(docs.confident-ai.com)

    1 points by jacky2wong 3y ago|0 comments
20. 80.▲
    How to unit test for LLM answer relevancy(docs.confident-ai.com)

    1 points by jacky2wong 3y ago|0 comments
21. 81.▲
    by jacky2wong 3y ago

    Got it - that wasn't immediately intuitive from the suggested comment. Yes! Our roadmap will include ways to test other parts of the LLM application stack very soon!
22. 82.▲
    by jacky2wong 3y ago

    A few publically shared evaluation datasets include: - HuggingFace Evaluate (and their 3 dashboards) - OpenAIEval - AnthropicEval
23. 83.▲
    by jacky2wong 3y ago

    Thanks for the interesting question! From what I've observed from both academia and posts from a number of top organizations (with seemingly the exception of OpenAI) - all the models are trained on test data. That's why OpenAI see
24. 84.▲
    by jacky2wong 3y ago

    Resolved all issues - thank you!
25. 85.▲
    by jacky2wong 3y ago

    Interesting - I'm not really sure how this would work but I'm happy to discuss this in more detail - I see this being as a suggestion for what to do when tests fail. Happy to discuss this via email if you'd like. Feel free to
26. 86.▲
    by jacky2wong 3y ago

    Thanks for the enormous amount of interest and questions in this post. I wanted to make a follow-up comment to answer all the questions above about differentiation, where we see issues in the current solution space and clarify the problem w
27. 87.▲
    by jacky2wong 3y ago

    See reply above^ :)
28. 88.▲
    by jacky2wong 3y ago

    Agreed. No one should ever have to touch YAML for writing unit tests for LLMs. Ever. Most people writing agents and LLM applications are Python developers/data scientists/ML engineers.
29. 89.▲
    by jacky2wong 3y ago

    Hi thanks for letting us know - please feel free to let us know of any more bugs - we'll make sure to write better docs and fix the code. We're just a few days in!
30. 90.▲
    by jacky2wong 3y ago

    Thanks. I've reviewed all of those solutions and happy to go into more detail in a later post. Here is our take: None of these solutions are built to fit seamlessly with the data team's workflow to collaborate and work together o