4 ms·
My sense is that these benchmarks are not realistic in terms of the way the model is used. People building specialized AI systems are not, in my experience, let
by andy99 1y ago
My sense is that these benchmarks are not realistic in terms of the way the model is used. People building specialized AI systems are not, in my experience, letting users just chat with a base model, they would have some variant of RAG plus some guardrails plus other stuff (like routing to pre-written answers for common question).
So what use case does this test setup reflect? Is there a relevant commercial use case here?
- dimitri-vs 1y agoYou could say the same for code benchmarks, no? For general medical Q&A I can't see how a specialized system would be better than base o3 with web search and a good prompt. If anything RAG and guardrail prompts would degrade performance.
- dontreact 1y agoIt tests people chatting to ChatGPT! That's a pretty big and important use case.