6 ms·
These days, lmsys elo is the only thing I trust. The other benchmark scores mean nothing to me at this point
by sujay1844 2y ago
These days, lmsys elo is the only thing I trust. The other benchmark scores mean nothing to me at this point
- __jl__ 2y agoI disagree. Not saying the other benchmarks are better. It just depends on your use case and application. For my use of the chat interface, I don't think lmsys is very useful. lmsys mainly evaluates relatively simple, low token count questions. Most (if not all) are single prompts, not conversations. The small models do well in this context. If that is what you are looking for, great. However, it does not test longer conversations with high token counts. Just saying that all benchmarks, including lmsys, have issues and are focused on specific use cases.