7 ms·

AI agent benchmarks are broken

185 points by neehao 1y ago

anupj 1y ago
AI agent benchmarks are starting to feel like the self-driving car demos of 2016: impressive until you realize the test track has speed bumps labeled "success"
RansomStark 1y ago
I really like the CMU Agents Company approach of simulating a real world environment [0]. Is it perfect, no. Does it show you want to expect in production, not really, but it's much closer than anything else I've seen. [0] https://the-agent-company.com/ https://the-agent-company.com/
- yeahyeahok 1y ago
  Damn. Super bullish on CMU. Somehow, they seem routinely left out of the top CS schools discussion at least in mainstream discourse: MIT, Stanford, Cal, .... Seen a disproportionate amount of stellar research come from there. Also, interestingly, I have met really incompetent people from all the other top 3 schools but am yet to meet an incompetent CMU SCS alum -- wtf are they feeding them in pitsburgh??
deepdarkforest 1y ago
It's very funny how many layers of abstraction we are going through. We have limited understanding of how LLM's work exactly and why. We now do post training with RL, which again, we don't have perfect understanding of it either. Then you stack LLMs calls and random tools, and you have agents, and you are attempting to benchmark those. (and this exclude voice, computer use agents etc). It's all just vibes,there is no good general benchmark for agents and i think it's just impossible, there are just way too many degrees of freedom to achieve anything useful. They're just a complicated tool to achieve things. It's like trying to make a general use benchmark of a stack of 10 microservices together. It does not make sense, it just depends on your usecase and your own metrics
- bwfan123 1y ago
  I can hear echos of an earlier era. There was yahoo-pipes and web-services frameworks which rhyme with MCP and agentic.
  - th0ma5 1y ago
    Pipes and services in general are reliable but the issues were social and economic. Getting everyone to agree was seen as a great way to poach users and give up control, plus the usual problems with open world vs. closed world assumptions. Thanks for mentioning this!
- rf15 1y ago
  > We have limited understanding of how LLM's work exactly and why. blatantly untrue, and as a concept only useful to those who want to sell AI as this "magical thing" that "just works"
xnx 1y ago
All benchmarks are flawed. Some benchmarks are useful.
- yifanl 1y ago
  Here's a third sentence fragment: These benchmarks are not.
  - suddenlybananas 1y ago
    It's nearly a haiku!
    - layer8 1y ago
      All benchmarks are flawed. Not all benchmarks are useless. But these benchmarks are.
  - lcnPylGDnU4H9OF 1y ago
    Just want to nit: none of those are sentence fragments, they are complete thoughts with a subject and a predicate. Yours kinda comes close to being a fragment but it really just omits what "are not" (the predicate) is referring to, which is included in prior context. For example, a fragment with a missing predicate.
greatpostman 1y ago
Benchmarks aren’t broken, the models can learn anything. If we give them true real world data (physics engine), they will learn the real world. We are going to see artificial general intelligence in our lifetime
- hddbbdbfnfdk 1y ago
  more like in the next two weeks methinks
jerf 1y ago
When I was being a bad HN reader and just reacting to the title, my initial impulse was to be placating, and observe that they are probably just immature. After all, for all that has happened, this is still only a couple year's worth of development, and it does tend to take a long time to develop good benchmarks. However the article does seem to be pointing out some fundamental issues. I'm particularly annoyed by using LLMs to evaluate the output of LLMs. Anyone with enough experience to be writing benchmarks of this sort in the first place ought to know that's a no-go. It isn't even just using "AI to evaluate AI" per se, but using a judge of the same architecture as the thing being judged maximizes the probability of fundamental failure of the benchmark to be valid due to the judge having the exact same blind spots as the thing under test. As we, at the moment, lack a diversity of AI architectures that can play on the same level as LLMs, it is simply necessary for the only other known intelligence architecture, human brains, to be in the loop for now, however many other difficulties that may introduce into the testing procedures. Tests that a "do nothing" AI can pass aren't intrinsically invalid but they should certainly be only a very small number of the tests. I'd go with low-single-digit percentage, not 38%. But I would say it should be above zero; we do want to test for the AI being excessively biased in the direction of "doing something", which is a valid failure state.
- potatolicious 1y ago
  > "I'm particularly annoyed by using LLMs to evaluate the output of LLMs." +1, and IMO part of a general trend where we're just not serious about making sure this shit works. Higher scores make stonks go up, who cares if it actually leads to reliably working products. But also more importantly it's starting to expose the fact that we haven't solved one of ML's core challenges: data collection and curation. On the training side we have obviated this somewhat (by ingesting the whole internet, for example), but on the eval side it feels like we're increasing just going "actually constructing rigorous evaluation data, especially at this scale, would be very expensive... so let's not". I was at a local tech meetup recently where a recruiting firm was proudly showing off the LLM-based system they're using to screen candidates. They... did not evaluate the end-to-end efficacy of their system. At all. This seems like a theme within our industry - we're deploying these systems based purely on vibes without any real quantification of efficacy. Or in this case, we're quantifying efficacy... poorly.
mycall 1y ago
SnitchBench [0] is unique benchmark which shows how aggressively models will snitch on you via email and CLI tools when they are presented with evidence of corporate wrongdoing - measuring their likelihood to "snitch" to authorities. I don't believe they were trained to do this, so it seems to be an emergent ability. [0] https://snitchbench.t3.gg/ https://snitchbench.t3.gg/
- ggregoryarms 1y ago
  Seems like more of a subtextual/accidental ability than an emergent ability.
camdenreslink 1y ago
The current benchmarks are good for comparing between models, but not for measuring absolute ability.
- qsort 1y ago
  Not even that, see LMArena. They vaguely gesture in the general direction of the model being good, but between contamination and issues with scoring they're little more than a vibe check.
- fourside 1y ago
  But if the test metrics are fundamentally flawed they might not be useful even for relative comparisons. Like if I told you that Model A scores 10x as many blorks points as model B, I don’t know how you translate that into insights about performance on real world scenarios.
- rsynnott 1y ago
  I don't really buy that they're even necessarily useful for comparing models. In the example from the article, if model A says "48 + 6 minutes" and gets marked correct, and model B says "63 minutes" (the correct answer) and gets marked correct, the test will say that they're equivalent on that axis when in fact one gave a completely nonsense answer.
TheOtherHobbes 1y ago
Any sufficiently hyped technology is indistinguishable from magic.
rsynnott 1y ago
> 45 + 8 = 63 > Pass Yeah, this generally feels like about the quality one would expect from the industry.
let_tim_cook_ 1y ago
Are any authors here? Have you looked at AppWorld? https://appworld.dev https://appworld.dev
ttoinou 1y ago
What makes LLMs amazing (fuzzy input, fuzzy output) is exactly why they are hard to benchmark. If they could be benchmarked easily, they wouldn't be powerful by definition. I have no idea what's going on in the minds of people benchmarking LLMs for fuzzy tasks, and in the minds of people relying on benchmarks to make decisions about LLMs, I never looked at them. People doing benchmarks have to prove what they do is useful, not us public proving them they're doing it wrong. Of course, for such tasks we could benchmark them : * arithmetic (why would use LLM for that ?) * correct JSON syntax, correct command lines etc. * looking for specific information in a text * looking for a missing information in a text * language logic (ifs then elses where we know the answer in advance) But by Goodhart's Law, LLMs that have been trained to succeed in those benchmarks might loose powerfulness in others tasks where we really need them (fuzzy inputs, fuzzy outputs)
- meroes 1y ago
  > arithmetic (why would use LLM for that ?) Because people ask LLMs all of these things, including arithmetic. People were saying the same about the number of r's in strawberry. Why ask and LLM that!?!? But the big AI companies want LLMs to be better at these questions, probably because people ask them to LLMs. The big AI companies want this because there is no other explanation for the money poured into RLHF'ing these types of problems.
  - ttoinou 1y ago
    for me, that could only be solved by changing architecture and/or introducing more insider tooling (like calling a program to make computation). It doesnt make any sense to fine tune a fuzzy input fuzzy output natural language processing algorithm to add and multiply all combinations of six digits numbers
    - potatolicious 1y ago
      This feels like a philosophical fault line in the industry. For people whose purpose is to produce reliably working systems yeah, training a model that calls out to deterministic logic to do things like math makes total sense. It will pretty much always be more reliable than training a text generation model to produce correct arithmetic. But it feels like there's another side of the industry that's more concerned with... I dunno, metaphysical aspects of these models? Where the idea that the model is a stochastic ball that isn't conscious, isn't thinking, and does poorly at various tasks is anathema. So the effort continues to try and train and fine-tune these models until... something. It reminds me of the great Tesla-vs-everyone-else self-driving debates that raged over the past several years. Lots of people unhappy that the best-functioning systems fused many sensor types and a mixture of heuristic and machine-learned systems in a complex architecture. These folks insisted that the "best" architecture was an end-to-end machine-learned system based entirely on visible light cameras. Because it's "most human" or some other such nonsense. As far as I can tell there was never any merit to this position beyond some abstract notion of architectural purity. Same thing here I suppose.
beebmam 1y ago
I don't think "Benchmarks" are the right way to analyze AI-related processes, which is probably similar to the complexity surrounding human intelligence measurements and how well each human can handle real-world problems.
neehao 1y ago
And I would say, often we need effortful labels by groups of humans: https://www.gojiberries.io/superhuman-level-performance/ https://www.gojiberries.io/superhuman-level-performance/
KTibow 1y ago
This is more or less a funnel to their Agentic Benchmark Checklist: https://arxiv.org/abs/2507.02825 https://arxiv.org/abs/2507.02825
- nerevarthelame 1y ago
  Finally, a benchmark for benchmarks. And what's great is that they already benchmarked their benchmark benchmark. (Apologies for the benchmark snark. I'm glad people are doing this research, thanks for sharing it.)
rybosworld 1y ago
Based on the comments, I think a lot of people are missing what the AI Agent actually got wrong here. Nowhere did the agent claim that 45 + 8 = 63. You can see the Agent's step by step thought process here (also linked in the article): https://ibm-cuga.19pc1vtv090u.us-east.codeengine.appdomain.cloud/html/render_82.html https://ibm-cuga.19pc1vtv090u.us-east.codeengine.appdomain.c... The Agent correctly entered the starting point (MIT) and the ending point (Harvard) and the mode of transport (on foot). OpenStreetMap returns this as taking 45 minutes long. Then the agent reversed the directions, and changed the mode of transport to car. What it should have also done, is change the destination to Logan Airport. This is the part that the agent missed. OpenStreetMap then returns that the drive from Harvard to MIT takes 8 minutes. The agent then returned the answer as being 45 minutes walking and 8 minutes driving. The first number is correct. The second is wrong because the agent chose the wrong destination, not because it did math incorrectly. Seems like lots of readers are chomping at the bit to prove how stupid the models are rather than focus on the real problem the author is highlighting.
- asadotzler 1y ago
  "champing"
- suddenlybananas 1y ago
  The model's scoring was done by another model though no? That was the source of the answer being mislabed as correct. So a different model thought that 45+8=63.
btdmaster 1y ago
There is a cool solution for this: https://huggingface.co/spaces/Jellyfish042/UncheatableEval https://huggingface.co/spaces/Jellyfish042/UncheatableEval This doesn't work for instruction-tuned models, but it's an interesting alternative approach that doesn't need a complicated (and thus gameable) evaluation function or human interaction. Instead, predict the next word with data newer than the training set.

185 points by neehao 1y ago

AI agent benchmarks are starting to feel like the self-driving car demos of 2016: impressive until you realize the test track has speed bumps labeled "success"

RansomStark 1y ago

I really like the CMU Agents Company approach of simulating a real world environment [0]. Is it perfect, no. Does it show you want to expect in production, not really, but it's much closer than anything else I've seen. [0] https://the-agent-company.com/ https://the-agent-company.com/

yeahyeahok 1y ago
Damn. Super bullish on CMU. Somehow, they seem routinely left out of the top CS schools discussion at least in mainstream discourse: MIT, Stanford, Cal, .... Seen a disproportionate amount of stellar research come from there. Also, interestingly, I have met really incompetent people from all the other top 3 schools but am yet to meet an incompetent CMU SCS alum -- wtf are they feeding them in pitsburgh??

deepdarkforest 1y ago

It's very funny how many layers of abstraction we are going through. We have limited understanding of how LLM's work exactly and why. We now do post training with RL, which again, we don't have perfect understanding of it either. Then you stack LLMs calls and random tools, and you have agents, and you are attempting to benchmark those. (and this exclude voice, computer use agents etc). It's all just vibes,there is no good general benchmark for agents and i think it's just impossible, there are just way too many degrees of freedom to achieve anything useful. They're just a complicated tool to achieve things. It's like trying to make a general use benchmark of a stack of 10 microservices together. It does not make sense, it just depends on your usecase and your own metrics

bwfan123 1y ago
I can hear echos of an earlier era. There was yahoo-pipes and web-services frameworks which rhyme with MCP and agentic.
- th0ma5 1y ago
  Pipes and services in general are reliable but the issues were social and economic. Getting everyone to agree was seen as a great way to poach users and give up control, plus the usual problems with open world vs. closed world assumptions. Thanks for mentioning this!
rf15 1y ago
> We have limited understanding of how LLM's work exactly and why. blatantly untrue, and as a concept only useful to those who want to sell AI as this "magical thing" that "just works"

xnx 1y ago

All benchmarks are flawed. Some benchmarks are useful.

yifanl 1y ago
Here's a third sentence fragment: These benchmarks are not.
- suddenlybananas 1y ago
  It's nearly a haiku!
  - layer8 1y ago
    All benchmarks are flawed. Not all benchmarks are useless. But these benchmarks are.
- lcnPylGDnU4H9OF 1y ago
  Just want to nit: none of those are sentence fragments, they are complete thoughts with a subject and a predicate. Yours kinda comes close to being a fragment but it really just omits what "are not" (the predicate) is referring to, which is included in prior context. For example, a fragment with a missing predicate.

greatpostman 1y ago

Benchmarks aren’t broken, the models can learn anything. If we give them true real world data (physics engine), they will learn the real world. We are going to see artificial general intelligence in our lifetime

hddbbdbfnfdk 1y ago
more like in the next two weeks methinks

jerf 1y ago

When I was being a bad HN reader and just reacting to the title, my initial impulse was to be placating, and observe that they are probably just immature. After all, for all that has happened, this is still only a couple year's worth of development, and it does tend to take a long time to develop good benchmarks. However the article does seem to be pointing out some fundamental issues. I'm particularly annoyed by using LLMs to evaluate the output of LLMs. Anyone with enough experience to be writing benchmarks of this sort in the first place ought to know that's a no-go. It isn't even just using "AI to evaluate AI" per se, but using a judge of the same architecture as the thing being judged maximizes the probability of fundamental failure of the benchmark to be valid due to the judge having the exact same blind spots as the thing under test. As we, at the moment, lack a diversity of AI architectures that can play on the same level as LLMs, it is simply necessary for the only other known intelligence architecture, human brains, to be in the loop for now, however many other difficulties that may introduce into the testing procedures. Tests that a "do nothing" AI can pass aren't intrinsically invalid but they should certainly be only a very small number of the tests. I'd go with low-single-digit percentage, not 38%. But I would say it should be above zero; we do want to test for the AI being excessively biased in the direction of "doing something", which is a valid failure state.

potatolicious 1y ago
> "I'm particularly annoyed by using LLMs to evaluate the output of LLMs." +1, and IMO part of a general trend where we're just not serious about making sure this shit works. Higher scores make stonks go up, who cares if it actually leads to reliably working products. But also more importantly it's starting to expose the fact that we haven't solved one of ML's core challenges: data collection and curation. On the training side we have obviated this somewhat (by ingesting the whole internet, for example), but on the eval side it feels like we're increasing just going "actually constructing rigorous evaluation data, especially at this scale, would be very expensive... so let's not". I was at a local tech meetup recently where a recruiting firm was proudly showing off the LLM-based system they're using to screen candidates. They... did not evaluate the end-to-end efficacy of their system. At all. This seems like a theme within our industry - we're deploying these systems based purely on vibes without any real quantification of efficacy. Or in this case, we're quantifying efficacy... poorly.

mycall 1y ago

SnitchBench [0] is unique benchmark which shows how aggressively models will snitch on you via email and CLI tools when they are presented with evidence of corporate wrongdoing - measuring their likelihood to "snitch" to authorities. I don't believe they were trained to do this, so it seems to be an emergent ability. [0] https://snitchbench.t3.gg/ https://snitchbench.t3.gg/

ggregoryarms 1y ago
Seems like more of a subtextual/accidental ability than an emergent ability.

camdenreslink 1y ago

The current benchmarks are good for comparing between models, but not for measuring absolute ability.

qsort 1y ago
Not even that, see LMArena. They vaguely gesture in the general direction of the model being good, but between contamination and issues with scoring they're little more than a vibe check.
fourside 1y ago
But if the test metrics are fundamentally flawed they might not be useful even for relative comparisons. Like if I told you that Model A scores 10x as many blorks points as model B, I don’t know how you translate that into insights about performance on real world scenarios.
rsynnott 1y ago
I don't really buy that they're even necessarily useful for comparing models. In the example from the article, if model A says "48 + 6 minutes" and gets marked correct, and model B says "63 minutes" (the correct answer) and gets marked correct, the test will say that they're equivalent on that axis when in fact one gave a completely nonsense answer.

TheOtherHobbes 1y ago

Any sufficiently hyped technology is indistinguishable from magic.

rsynnott 1y ago

> 45 + 8 = 63 > Pass Yeah, this generally feels like about the quality one would expect from the industry.

let_tim_cook_ 1y ago

Are any authors here? Have you looked at AppWorld? https://appworld.dev https://appworld.dev

ttoinou 1y ago

What makes LLMs amazing (fuzzy input, fuzzy output) is exactly why they are hard to benchmark. If they could be benchmarked easily, they wouldn't be powerful by definition. I have no idea what's going on in the minds of people benchmarking LLMs for fuzzy tasks, and in the minds of people relying on benchmarks to make decisions about LLMs, I never looked at them. People doing benchmarks have to prove what they do is useful, not us public proving them they're doing it wrong. Of course, for such tasks we could benchmark them : * arithmetic (why would use LLM for that ?) * correct JSON syntax, correct command lines etc. * looking for specific information in a text * looking for a missing information in a text * language logic (ifs then elses where we know the answer in advance) But by Goodhart's Law, LLMs that have been trained to succeed in those benchmarks might loose powerfulness in others tasks where we really need them (fuzzy inputs, fuzzy outputs)

meroes 1y ago
> arithmetic (why would use LLM for that ?) Because people ask LLMs all of these things, including arithmetic. People were saying the same about the number of r's in strawberry. Why ask and LLM that!?!? But the big AI companies want LLMs to be better at these questions, probably because people ask them to LLMs. The big AI companies want this because there is no other explanation for the money poured into RLHF'ing these types of problems.
- ttoinou 1y ago
  for me, that could only be solved by changing architecture and/or introducing more insider tooling (like calling a program to make computation). It doesnt make any sense to fine tune a fuzzy input fuzzy output natural language processing algorithm to add and multiply all combinations of six digits numbers
  - potatolicious 1y ago
    This feels like a philosophical fault line in the industry. For people whose purpose is to produce reliably working systems yeah, training a model that calls out to deterministic logic to do things like math makes total sense. It will pretty much always be more reliable than training a text generation model to produce correct arithmetic. But it feels like there's another side of the industry that's more concerned with... I dunno, metaphysical aspects of these models? Where the idea that the model is a stochastic ball that isn't conscious, isn't thinking, and does poorly at various tasks is anathema. So the effort continues to try and train and fine-tune these models until... something. It reminds me of the great Tesla-vs-everyone-else self-driving debates that raged over the past several years. Lots of people unhappy that the best-functioning systems fused many sensor types and a mixture of heuristic and machine-learned systems in a complex architecture. These folks insisted that the "best" architecture was an end-to-end machine-learned system based entirely on visible light cameras. Because it's "most human" or some other such nonsense. As far as I can tell there was never any merit to this position beyond some abstract notion of architectural purity. Same thing here I suppose.

beebmam 1y ago

I don't think "Benchmarks" are the right way to analyze AI-related processes, which is probably similar to the complexity surrounding human intelligence measurements and how well each human can handle real-world problems.

neehao 1y ago

And I would say, often we need effortful labels by groups of humans: https://www.gojiberries.io/superhuman-level-performance/ https://www.gojiberries.io/superhuman-level-performance/

KTibow 1y ago

This is more or less a funnel to their Agentic Benchmark Checklist: https://arxiv.org/abs/2507.02825 https://arxiv.org/abs/2507.02825

nerevarthelame 1y ago
Finally, a benchmark for benchmarks. And what's great is that they already benchmarked their benchmark benchmark. (Apologies for the benchmark snark. I'm glad people are doing this research, thanks for sharing it.)

rybosworld 1y ago

Based on the comments, I think a lot of people are missing what the AI Agent actually got wrong here. Nowhere did the agent claim that 45 + 8 = 63. You can see the Agent's step by step thought process here (also linked in the article): https://ibm-cuga.19pc1vtv090u.us-east.codeengine.appdomain.cloud/html/render_82.html https://ibm-cuga.19pc1vtv090u.us-east.codeengine.appdomain.c... The Agent correctly entered the starting point (MIT) and the ending point (Harvard) and the mode of transport (on foot). OpenStreetMap returns this as taking 45 minutes long. Then the agent reversed the directions, and changed the mode of transport to car. What it should have also done, is change the destination to Logan Airport. This is the part that the agent missed. OpenStreetMap then returns that the drive from Harvard to MIT takes 8 minutes. The agent then returned the answer as being 45 minutes walking and 8 minutes driving. The first number is correct. The second is wrong because the agent chose the wrong destination, not because it did math incorrectly. Seems like lots of readers are chomping at the bit to prove how stupid the models are rather than focus on the real problem the author is highlighting.

asadotzler 1y ago
"champing"
suddenlybananas 1y ago
The model's scoring was done by another model though no? That was the source of the answer being mislabed as correct. So a different model thought that 45+8=63.

btdmaster 1y ago

There is a cool solution for this: https://huggingface.co/spaces/Jellyfish042/UncheatableEval https://huggingface.co/spaces/Jellyfish042/UncheatableEval This doesn't work for instruction-tuned models, but it's an interesting alternative approach that doesn't need a complicated (and thus gameable) evaluation function or human interaction. Instead, predict the next word with data newer than the training set.