Benchmarks are often weird because of what a benchmark inherently needs to be. If you compare LLMs by asking them to tell you how to catch dragonflies - the fr

Benchmarks are often weird because of what a benchmark inherently needs to be. If you compare LLMs by asking them to tell you how to catch dragonflies - the free text chat answer you get will be impossible to objectively evaluate. Whereas if you propose four ways to catch dragonflies and ask each model to choose option A, B, C or D (or check the relative probability the model assigns to those four output logits) the result is easy to objectively evaluate - you just check if it chose the one right answer. Hence a lot of the most famous benchmarks are multiple-choice questions - even though 99.9% of LLM usage doesn't involve answering multiple-choice questions.