3 ms·
Arena lets you use 1k out of those 128k available tokens. It's not a good test.
by irthomasthomas 2y ago
Arena lets you use 1k out of those 128k available tokens. It's not a good test.
- moffkalast 2y agoWell they only have so much compute right? Beats the usual multiple choice, one token output benchmarks.
- causal 2y agoIt's a human-preference benchmark. Useful but not sufficient.