4 ms·

> ...but no. They do not beat opus on real-world usage. I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving

> ...but no. They do not beat opus on real-world usage. I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.

bewareofscams 2mo ago
Only useful benchmarks are those you (in particular) don't have access to.
- rhdunn 2mo ago
  The only useful benchmarks are those you've created for your specific workflow. Only then can you assess whether a given model is better or worse for what you are using it for. There are tools like promptfoo designed for this.
niek_pas 2mo ago
A wise man once said, "not everything that counts can be counted, and not everything that can be counted counts".
- valvar 2mo ago
  That’s well and good, but how are we supposed to evaluate the accuracy of random HN comments without anything resembling somewhat objective metrics? People say all manner of things, and usually it’s contradictory. What heuristic do you propose?
xienze 2mo ago
> but then we just need meaningful benchmarks that clearly show that! That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria. It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."
pimeys 2mo ago
If you are working in a company and using language models, it is a very good idea to hold a bunch of evals you can trust and use to validate new models. Calibrate every once in a while with prod data. We have our own and the only numbers on quality and cost I trust come from this setup.
mlmonkey 2mo ago
In the end, the only benchmark that matters is your own.

bewareofscams 2mo ago

Only useful benchmarks are those you (in particular) don't have access to.

rhdunn 2mo ago
The only useful benchmarks are those you've created for your specific workflow. Only then can you assess whether a given model is better or worse for what you are using it for. There are tools like promptfoo designed for this.

niek_pas 2mo ago

A wise man once said, "not everything that counts can be counted, and not everything that can be counted counts".

valvar 2mo ago
That’s well and good, but how are we supposed to evaluate the accuracy of random HN comments without anything resembling somewhat objective metrics? People say all manner of things, and usually it’s contradictory. What heuristic do you propose?

xienze 2mo ago

> but then we just need meaningful benchmarks that clearly show that! That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria. It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."

pimeys 2mo ago

If you are working in a company and using language models, it is a very good idea to hold a bunch of evals you can trust and use to validate new models. Calibrate every once in a while with prod data. We have our own and the only numbers on quality and cost I trust come from this setup.

mlmonkey 2mo ago

In the end, the only benchmark that matters is your own.