3 ms·
> So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? No, we don't. LMsys is just one, very flawed benchmark
by zone411 2y ago
> So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones?
No, we don't. LMsys is just one, very flawed benchmark.
- oersted 2y agoAgreed, but it's wild that even one benchmark shows this. Based on what we knew just a few months ago, these models should be so far from each other in every benchmark.
- ukuina 2y agoWhy is LMsys flawed? Many people treat LMsys as gospel because it's the only large-scale, up-to-date qualitative benchmark. All the numeric benchmarks seem to miss real-world applicability.