4 ms·

AI Capabilities May Be Overhyped on Bogus Benchmarks, Study Finds

43 points by Cynddl 11mo ago

lispisok 11mo ago
There is way too much money being thrown at AI to not game/cheat the benchmarks
vivzkestrel 11mo ago
I am amazed not a single pro AI person on HN has anything to say or even speculate about this. This is such a serious issue
- ulfw 11mo ago
  Because the pro AI persons are busy trying sell their whatevertheyhave before the bubble bursts
- simianwords 11mo ago
  This is a very poor article. What I understood is that they take one benchmark (in particular) that tests grade school level math. This benchmark apparently claims to test ability to reason through math problems. They agree that the benchmarks show that the LLMs can solve such questions and models are getting better. But their main point is that this does not prove that the model is reasoning. But so what??? It may not reason in the way humans do but it is pretty damn close. The mechanics are the same - recursively generate a prompt that terminates in an answer generating prompt. They don’t like that this indicates the model “reasons through” the problem. But it’s just semantics at this point. For me and for most others - getting the final answer is important. And it largely accomplishes this task. I don’t buy that the model couldn’t reason through - have you ever asked a model for its explanation? It does genuinely explain how it got the solution. At this point who the hell cares what “reasoning” means if it 1. Gets me the right answer 2. Reasonably explains how it did it
  - YeGoblynQueenne 11mo ago
    We care whether it's reasoning or not because the alternative is that it's guessing, rather than reasoning, and when guessing is measured on benchmarks that are supposed to measure reasoning, the results are likely to be misleading. Why do we care if the benchmark results are misleading? The reason we have benchmarks in machine learning is that we can use the results on the benchmarks to predict the performance of a system in uncontrolled conditions, i.e. "in the real world". If the benchmarks don't measure what we think they measure then they can't be used to make that kind of prediction. If that's the case then we really have no idea how good or bad a system really is. Seen another way, if a benchmark is not measuring what we think it measures, all we learn from a system passing the benchmark is that the system passes the benchmark. Still, what do you care if it gets you the right answer? The question is, exactly, how do you know it's really getting you the right answer? Maybe you can tell when you know the answer, but what about answers you genuinely don't know? And how often does it get you the wrong answer but you don't realise? You can't realistically test an AI system by interacting with it as thoroughly and as rigorously as you can with... a benchmark. That's why we care about having accurate benchmarks that measure what they're supposed to be measuring. P.S. Another issue of course is that guessing is limited while reasoning is... less limited. We care about reasoning because we ideally want to have systems that are better than the best guessing machine.
simianwords 11mo ago
“ When researchers tested the same performance on a new set of benchmark questions, they noticed that models experienced “significant performance drops.”” This is very misleading because the generalisation ability of LLMs is very very high. It doesn’t just memorise problems - that’s nonsense. At high school level maths you genuinely can’t get gpt-5 thinking to make a single mistake. Not possible at all. Unless you give some convoluted ambiguous prompt that no human can understand. If you assume I’m correct, how does gpt memorise then? In fact even undergraduate level mathematics is quite simple for gpt-5 thinking. IMO gold was won.. by what? Memorising solutions? I challenge people to find ONE example that gpt-5 thinking gets wrong in high school or undergrad level maths. I could not achieve it. You must allow all tools though.
- callmesnek 11mo ago
  "You must allow all tools though"
  - y0eswddl 11mo ago
    "if you don't let chatgpt search the internet or use python code, it doesn't count..." look at those goalposts go!
    - simianwords 11mo ago
      Ok don’t allow search or python. Can you come up with an example? Probably not.
- YeGoblynQueenne 11mo ago
  The best performance on GSM8K is currently at 0.973, so less than perfect [1]. Given that GSM8K is a grade school math question data set, and the leading LLMs still don't get all answers correctly it's safe to assume that they won't get all high school questions' answers correctly either, since those are going to be harder than grade school questions. This means there has got to be at least one example that GPT-5 as well as every other LLM fails on [2]. If you don't think that's the case I think it's up to you to show that it's not. ___________________ [1] GSM8K leaderboard: https://llm-stats.com/benchmarks/gsm8k https://llm-stats.com/benchmarks/gsm8k [2] This is regardless of what GSM8K or any other benchmark is measuring.
Khaine 11mo ago
I'm shocked, shocked, that AI is optimised to pass bogus benchmarks. Just like how GPUs were optimised to pass synthetic benchmarks.

43 points by Cynddl 11mo ago

There is way too much money being thrown at AI to not game/cheat the benchmarks

I am amazed not a single pro AI person on HN has anything to say or even speculate about this. This is such a serious issue

ulfw 11mo ago
Because the pro AI persons are busy trying sell their whatevertheyhave before the bubble bursts
simianwords 11mo ago
This is a very poor article. What I understood is that they take one benchmark (in particular) that tests grade school level math. This benchmark apparently claims to test ability to reason through math problems. They agree that the benchmarks show that the LLMs can solve such questions and models are getting better. But their main point is that this does not prove that the model is reasoning. But so what??? It may not reason in the way humans do but it is pretty damn close. The mechanics are the same - recursively generate a prompt that terminates in an answer generating prompt. They don’t like that this indicates the model “reasons through” the problem. But it’s just semantics at this point. For me and for most others - getting the final answer is important. And it largely accomplishes this task. I don’t buy that the model couldn’t reason through - have you ever asked a model for its explanation? It does genuinely explain how it got the solution. At this point who the hell cares what “reasoning” means if it 1. Gets me the right answer 2. Reasonably explains how it did it
- YeGoblynQueenne 11mo ago
  We care whether it's reasoning or not because the alternative is that it's guessing, rather than reasoning, and when guessing is measured on benchmarks that are supposed to measure reasoning, the results are likely to be misleading. Why do we care if the benchmark results are misleading? The reason we have benchmarks in machine learning is that we can use the results on the benchmarks to predict the performance of a system in uncontrolled conditions, i.e. "in the real world". If the benchmarks don't measure what we think they measure then they can't be used to make that kind of prediction. If that's the case then we really have no idea how good or bad a system really is. Seen another way, if a benchmark is not measuring what we think it measures, all we learn from a system passing the benchmark is that the system passes the benchmark. Still, what do you care if it gets you the right answer? The question is, exactly, how do you know it's really getting you the right answer? Maybe you can tell when you know the answer, but what about answers you genuinely don't know? And how often does it get you the wrong answer but you don't realise? You can't realistically test an AI system by interacting with it as thoroughly and as rigorously as you can with... a benchmark. That's why we care about having accurate benchmarks that measure what they're supposed to be measuring. P.S. Another issue of course is that guessing is limited while reasoning is... less limited. We care about reasoning because we ideally want to have systems that are better than the best guessing machine.

simianwords 11mo ago

“ When researchers tested the same performance on a new set of benchmark questions, they noticed that models experienced “significant performance drops.”” This is very misleading because the generalisation ability of LLMs is very very high. It doesn’t just memorise problems - that’s nonsense. At high school level maths you genuinely can’t get gpt-5 thinking to make a single mistake. Not possible at all. Unless you give some convoluted ambiguous prompt that no human can understand. If you assume I’m correct, how does gpt memorise then? In fact even undergraduate level mathematics is quite simple for gpt-5 thinking. IMO gold was won.. by what? Memorising solutions? I challenge people to find ONE example that gpt-5 thinking gets wrong in high school or undergrad level maths. I could not achieve it. You must allow all tools though.

callmesnek 11mo ago
"You must allow all tools though"
- y0eswddl 11mo ago
  "if you don't let chatgpt search the internet or use python code, it doesn't count..." look at those goalposts go!
  - simianwords 11mo ago
    Ok don’t allow search or python. Can you come up with an example? Probably not.
YeGoblynQueenne 11mo ago
The best performance on GSM8K is currently at 0.973, so less than perfect [1]. Given that GSM8K is a grade school math question data set, and the leading LLMs still don't get all answers correctly it's safe to assume that they won't get all high school questions' answers correctly either, since those are going to be harder than grade school questions. This means there has got to be at least one example that GPT-5 as well as every other LLM fails on [2]. If you don't think that's the case I think it's up to you to show that it's not. ___________________ [1] GSM8K leaderboard: https://llm-stats.com/benchmarks/gsm8k https://llm-stats.com/benchmarks/gsm8k [2] This is regardless of what GSM8K or any other benchmark is measuring.

Khaine 11mo ago

I'm shocked, shocked, that AI is optimised to pass bogus benchmarks. Just like how GPUs were optimised to pass synthetic benchmarks.