4 ms·
lots of attempts at the llm eval game: - https://github.com/BerriAI/bettertest https://github.com/BerriAI/bettertest https://twitter.com/ishaan_jaff/status/166
by swyx 3y ago
lots of attempts at the llm eval game:
- https://github.com/BerriAI/bettertest https://github.com/BerriAI/bettertest https://twitter.com/ishaan_jaff/status/1665105582804832258 https://twitter.com/ishaan_jaff/status/1665105582804832258
- https://github.com/AgentOps-AI/agentops https://github.com/AgentOps-AI/agentops
- https://www.ycombinator.com/launches/JFc-baserun-ai-ship-llm-features-with-confidence https://www.ycombinator.com/launches/JFc-baserun-ai-ship-llm...
- https://news.ycombinator.com/item?id=36958175 https://news.ycombinator.com/item?id=36958175
- https://github.com/promptfoo/promptfoo https://github.com/promptfoo/promptfoo
can you articulate what insight on the market you have that will make you stand out over time?
- alook 3y agoAlso would love to hear an answer to this
- jacky2wong 3y agoSee reply above^ :)
- azzarcher 3y agoDon't forget https://benchllm.com/ https://benchllm.com/
- swyx 3y agoappreciate it - part of why i put this list up is so that people can add to it lol
- Areibman 3y agoPlenty more: - https://www.getscorecard.ai https://www.getscorecard.ai - https://arxiv.org/abs/2308.03688 https://arxiv.org/abs/2308.03688 - https://withmartian.com https://withmartian.com - https://aihero.studio/ https://aihero.studio/ (author of agentops here)
- swyx 3y agoadded to my notes! https://github.com/swyxio/ai-notes/ https://github.com/swyxio/ai-notes/
- jacky2wong 3y agoThanks. I've reviewed all of those solutions and happy to go into more detail in a later post. Here is our take: None of these solutions are built to fit seamlessly with the data team's workflow to collaborate and work together on (imagine the difference of Modal and WandB had on data / ML teams). We have built and consulted on a number of LLM projects and found that prototypes were super easy to make. Iteration was incredibly difficult. Our goal was to make iteration as easy as writing a unit test. A number of these projects are missing most Python developer's workflow. For example - we made it compatible with Pytest on day 1 because that's the most common testing suite. Many competitor tools all make a good pass at these but our roadmap is significantly different and not catered for normal web developers etc. Our focus on building this is to ensure the first step makes sense - that we are super intuitive to add to a data team's Python workflow. (Just 1 assert statement to write a unit test).
- swyx 3y agocool. thanks for answering!
- resiros 3y agoI'd add ours too, although we're trying to be an end-to-end one-stop platform. https://github.com/agenta-ai/agenta https://github.com/agenta-ai/agenta
- swyx 3y agoyou and everyone else, unfortunately. gonna be brutal out there and take exceptional entrepreneurship to be the actual end to end platform that wins
- d4rkp4ttern 3y agoThis is a fair question on the surface, but one thing that occurs to me — given the sheer number of similar-looking things people put out, people may not have the time to exhaustively compare their offering with others. You often only discover differentiation by diving deep into each framework/lib etc, and if you dive into every Twitter-amplified new framework/package then you would never have time build your own thing. So I guess what is a builder to do? Probably focus on practical use cases, show adoption and let these speak for themselves?
- deleted 3y ago[deleted]
- swyx 3y agofor what its worth, i agree, i've had this exact chat with @areibman (who is replying downthread). this is why i asked for market insight rather than a competitor by competitor diff.
- d4rkp4ttern 3y agoGot it, that makes sense
- fswd 3y agoAre there any large or publically shared evaluation datasets and corresponding question and answer validation data?
- jacky2wong 3y agoA few publically shared evaluation datasets include: - HuggingFace Evaluate (and their 3 dashboards) - OpenAIEval - AnthropicEval
- intended 3y agoI would not use those for production. I would assume your backend model is already tested on these. You would need a novel data set.