3 ms·
Congratulations on the launch! I was unable to use this library: I was trying to evaluate different non-OpenAI models and it consistently failed due to malform
by jfisher4024 3y ago
Congratulations on the launch! I was unable to use this library: I was trying to evaluate different non-OpenAI models and it consistently failed due to malformed JSONs coming from the model.
Any thoughts about using different models? Is this just a langchain limitation?
- shahules 3y agoThanks for your feedback. We have tested Ragas on alternatives like Claude, Mixtral, Gemini, etc. Although we support all LLMs supported by Langchain, sadly many of the OSS models out of the box aren't capable of generating JSON-type output which is important for us to ensure reproducibility.
- jfisher4024 3y agoAny tips for Mixtral? That’s what we tried
- shahules 3y agoHey, I would recommend checking out our PRs. There would be PRs that have modified some of the prompts to better suit Mixtral.
- 3d27 3y agoCheckout this instead: https://github.com/confident-ai/deepeval https://github.com/confident-ai/deepeval Also has native ragas implementation but supports all models.