4 ms·
Ask HN: What are you using for LLM response testing and benchmarking?
What are you using to test your LLM responses, benchmark them, maybe compare different versions?
I've seen a few YC startups focusing on this but I haven't decided yet if we should build this internally or use an external tool.
- muzani 3y agoThis list has been nice: https://lmsys.org/blog/2023-05-25-leaderboard/ https://lmsys.org/blog/2023-05-25-leaderboard/ And you can participate in the arena, which pits them against each other. I'm surprised that I actually voted for GPT-3.5 over GPT-4 for a lot of my use cases.
- tikkun 3y agoWhat features are critical for you in your response testing and benchmarking? I think it'll be easier to answer with specifics.
- tin7in 3y agoWe are using chat/completion APIs and we have a set of prompts that users can apply on selected text. I would like to see how changing the system prompt or any of the logic in the pre-defined prompts affects the output.
- tikkun 3y agoCould you let me know if one of these is what you're looking for? I put this list together, I'm pretty sure one of these should solve what you're after: https://llm-utils.org/List+of+tools+for+prompt+engineering https://llm-utils.org/List+of+tools+for+prompt+engineering
- tin7in 3y agoThis is great, thank you! Promptclub and ChainForge look exactly what I need.