Not really. They already chose to show the benchmark where it does best and even then it’s still quite a bit worse (though definitely impressive for its size).

Not really. They already chose to show the benchmark where it does best and even then it’s still quite a bit worse (though definitely impressive for its size). If you take a look at other benchmarks, for example MMLU@5-shot then this does 46.3, while gpt-3.5 does 70. But there might be some use cases where this one is close enough in performance and the difference in cost and speed make it a better choice.