8 ms·

Study identifies weaknesses in how AI systems are evaluated

416 points by pseudolus 1y ago

Paper: https://openreview.net/pdf?id=mdA5lVvNcU https://openreview.net/pdf?id=mdA5lVvNcU Related: https://www.theregister.com/2025/11/07/measuring_ai_models_hampered_by/ https://www.theregister.com/2025/11/07/measuring_ai_models_h...

Marshferm 1y ago
Don’t get high on your own supply.
calpaterson 1y ago
Definitely one of the weaker areas in the current LLM boom. Comparing models, or even different versions of the same model, is a pseudo-scientific mess. I'm still using https://lmarena.ai/leaderboard https://lmarena.ai/leaderboard. Perhaps there is something better and someone will pipe up to tell me about it. But we use LLMs at work and have unexplainable variations between them. And when we get a prompt working reliably on one model, we often have trouble porting it to another LLM - even straight "version upgrades" such as from GPT-4 to -5. Your prompt and your model become highly coupled quite easily. I dunno what to do about it and am tending to just pick Gemini as a result.
- HPsquared 1y ago
  Psychometric testing of humans has a lot of difficulties, too. It's hard to measure some things.
- ACCount37 1y ago
  Ratings on LMArena are too easily gamed. Even professional human evaluators are quite vulnerable to sycophancy and overconfident-and-wrong answers. And LMArena evaluators aren't professionals. A lot of the sycophancy mess that seeps from this generation of LLM stems from reckless tuning based on human feedback. Tuning for good LMArena performance has similar effects - and not at all by a coincidence.
  - energy123 1y ago
    It's biased to small context performance, which is why I don't pay much attention to it as a developer aside from a quick glance. I need performance at 40-100k tokens which models like Deepseek can't deliver but Gemini 2.5 Pro and ChatGPT 5.0 Thinking can.
    - ACCount37 1y ago
      And even "long term performance" splits itself into "performance on multi-turn instruction following" and "performance on agentic tasks" down the line. And "performance on agentic tasks" is a hydra in itself. Capturing LLM performance with a single metric is a hopeless task. But even a single flawed metric beats no metrics at all.
- diamond559 1y ago
  I'd rather quit then be forced to beta test idiocracy. What's your company so we can all avoid it?
- botro 1y ago
  This is something I've stuggled with for my site, I made https://aimodelreview.com/ https://aimodelreview.com/ to compare the outputs of LLMs over a variety of prompts and categories, allowing a side by side comparison between them. I ran each prompt 4 times for each model with different temperature values available as a toggles. My thinking was to just make the responses available to users and let them see how models perform. But from some feedback, turns out users don't want to have to evaluate the answers and would rather see a leaderboard and rankings. The scalable solution to that would be LLM as judge that some benchmarks already use, but that just feels wrong to me. LM Arena tries to solve this with the crowd sourced solution, but I think the right method would have to be domain expert human reviewers, so like Wirecutter VS IMDb, but that is expensive to pull off.
- 3abiton 1y ago
  > Comparing models, or even different versions of the same model, is a pseudo-scientific mess. Reminder that in most cases, it's impossible to know if there is cross-contamination from the test set of the public benchmarks, as most LLMs are not truely open-source. We can't replicate them. So arguably it's worse in some cases, pretty much fraud if you account for the VC money pouring in. This is even more evident in unknown models from lesser known institutes like from UAE.
- andai 1y ago
  >when we get a prompt working reliably on one model, we often have trouble porting it to another LLM I saw a study where a prompt massively boosted one model's performance on a task, but significantly reduced another popular model's performance on the same task.
  - boccaff 1y ago
    Do you have any pointer to search for that?
bubblelicious 1y ago
I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster. No one really has a good solution, and researchers are also in a huge rush and don’t want to end up making their whole job benchmarking. Even if you could, and even if you have the right background you can do benchmarks full time and they still would be a mess. Product testing (with traditional A/B tests) are kind of the best bet since you can measure what you care about _directly_ and at scale. I would say there is of course “benchmarketing” but generally people do sincerely want to make good benchmarks it’s just hard or impossible. For many of these problems we’re hitting capabilities where we don’t even have a decent paradigm to use,
- ACCount37 1y ago
  A/B testing is radioactive too. It's indirectly optimizing for user feedback - less stupid than directly optimizing for user feedback, but still quite dangerous. Human raters are exploitable, and you never know whether the B has a genuine performance advantage over A, or just found a meat exploit by an accident. It's what fucked OpenAI over with 4o, and fucked over many other labs in more subtle ways.
  - bubblelicious 1y ago
    Are you talking about just preferences or A/B tests on like retention and engagement? The latter I think is pretty reliable and powerful though I have never personally done them. Preferences are just as big a mess: WHO the annotators are matters, and if you are using preferences as a proxy for like correctness, you’re not really measuring correctness you’re measuring e.g. persuasion. A lot of construct validity challenges (which themselves are hard to even measure in domain).
    - ACCount37 1y ago
      Yes. All of them are poisoned metrics, just in different ways. GPT-4o's endless sycophancy was great for retention, GPT-5's style of ending every response in a question is great for engagement. Are those desirable traits though? Doubt it. They look like simple tricks and reek of reward hacking - and A/B testing rewards them indeed. Direct optimization is even worse. Combining the two is ruinous. Mind, I'm not saying that those metrics are useless. Radioactive materials aren't useless. You just got to keep their unpleasant properties in mind at all times - or suffer the consequences.
jennyholzer 1y ago
I've been getting flagged by high-on-their-own-supply AI boosters for identifying that LLM benchmarks have been obvious bullshit for at least the last year and a half. What changed to make "the inevitable AI bubble" the dominant narrative in last week or so?
- conception 1y ago
  Companies are talking about needing trillions of dollars is why.
  - foobiekr 1y ago
    And the government backstops. Nothing says confidence that AGI is imminent like needing the US government to prevent your investments from losing you money.
- HPsquared 1y ago
  Benchmarks in general have this problem, across pretty much all industries. "When a measure becomes a target" and all that.
- icameron 1y ago
  The market was down for AI related stocks especially, while down only over 3% it’s the worst week since April, and there’s no single event that is to blame it just looks like market sentiment has shifted away from the previous unchecked exuberance.
- Kiro 1y ago
  Link those comments please because I checked your history and the flagged ones were pure nonsense with zero insights. Also, calling out LLM benchmarks has never been a radical take and basically the default on this site.
- purple_turtle 1y ago
  It is possible to be right on the main theme but only by accident (with arguments and claims being wrong), communicating in a highly faulty way, with pointless insults, doing it in offtopic derails, being correct on minor point while being mostly wrong etc. Can you link some of these comments you consider useful but got flagged?
mikert89 1y ago
[flagged]
- jennyholzer 1y ago
  [flagged]
- ZyanWu 1y ago
  Any argument to back up your claim?
  - HPsquared 1y ago
    The fact that I could type this exchange into an LLM and get a list of reasonable arguments.
  - CooCooCaCha 1y ago
    Not the person you replied to, but I'm a programmer and up until ~3 weeks ago I really only used AI for auto-complete, looking up API information, and constructing arcane CLI commands. I decided to take a leap and use AI as much as possible to complete a ticket at work. Now, 3 weeks later AI is writing 90% of my code. Granted, I'm not sitting back sipping on a latte while AI does my job. It's a very interactive process, I spend more time reviewing code and going back and forth with the AI to get the result I want. But it's become surprisingly good. I wouldn't say I'm anywhere close to 10x more productive, but perhaps 50%.
    - GPerson 1y ago
      It definitely has its amazing moments, but sometimes I get caught in a loop of expecting it to do the thing, it not working, and spinning my wheels a lot instead of just solving it myself. I think I’m still learning how to use the tools effectively, but the random nature of it makes it difficult.
    - diamond559 1y ago
      Yeah, nobody believes your made up numbers buddy. You're a stonk pumper, we know it.
shanev 1y ago
This is solvable at the level of an individual developer. Write your own benchmark for code problems that you've solved. Verify tests pass and that it satisfies your metrics like tok/s and TTFT. Create a harness that works with API keys or local models (if you're going that route).
- motoboi 1y ago
  Well, openai github is open to write evaluations. Just add your there and guaranteed that the next model will perform better on them.
- cactusplant7374 1y ago
  I think that's what this site is doing: https://aistupidlevel.info/ https://aistupidlevel.info/
- davedx 1y ago
  That’s called evals and yes any serious AI project uses them
- hamdingers 1y ago
  At the developer level all my LLM use is in the context of agentic wrappers, so my benchmark is fairly trivial: Configure aider or claude code to use the new model, try to do some work. The benchmark is pass/fail, if after a little while I feel the performance is better than the last model I was using it's a pass, otherwise it's a fail and I go back. Building your own evaluations makes sense if you're serving an LLM up to customers and want to know how it performs, but if you are the user... use it and see how it goes. It's all subjective anyway.
  - embedding-shape 1y ago
    > Building your own evaluations makes sense if you're serving an LLM up to customers and want to know how it performs, but if you are the user... use it and see how it goes. It's all subjective anyway. I'd really caution against this approach, mainly because humans suck at removing emotions and other "human" factors when judging how well something works, but also because comparing across models gets a lot easier when you can see 77/100 vs 91/100 as a percentage score, over your own tasks that you actually use the LLMs for. Just don't share this benchmark publicly once you're using it for measurements.
bee_rider 1y ago
> "For example, if a benchmark reuses questions from a calculator-free exam such as AIME," the study says, "numbers in each problem will have been chosen to facilitate basic arithmetic. Testing only on these problems would not predict performance on larger numbers, where LLMs struggle." When models figure out how to exploit an effect that every clever college student does, that should count as a win. That’s a much more human-like reasoning ability, than the ability to multiply large numbers or whatever (computers were already good at that, to the point that it has become a useless skill for humans to have). The point of these LLMs is to do things that computers were bad at.
- layer8 1y ago
  I don’t think the fact that LLMs can handle small numbers more reliably has anything to do with their reasoning ability. To the contrary, reasoning ability should enable them to handle numbers of arbitrary size, just as it enables humans to do so, given some pencil and paper. However: > Testing only on these problems would not predict performance on larger numbers, where LLMs struggle. Since performance on large numbers is not what these exams are intended to test for, I don’t see this as a counterargument, unless the benchmarks are misrepresenting what is being tested for.
  - zamadatix 1y ago
    Pencil and paper is just testing with tools enabled.
    - layer8 1y ago
      You seem to be addressing an argument that wasn’t made. Personally, I’d say that such tool use is more akin to a human using a calculator.
      - zamadatix 1y ago
        I'm not addressing an argument, just stating that's already a form of LLM testing done today for people wanting to look at the difference in results the same as the human analogy.
moritzwarhier 1y ago
When people claim that there is such a thing as "X% accuracy in reasoning", it's really hard to take anything else seriously, no matter how impressive. AI (and humans!) aside, claiming that there was an oracle that could "answer all questions" is a solved problem. Such a thing cannot exist. But this is going already too deep IMO. When people start talking about percentages or benchmark scores, there has to be some denominator. And there can be no bias-free such denominator for - trivia questions - mathematical questions (oh, maybe I'm wrong here, intuitively I'd say it's impossible for various reasons: varying "hardness", undecidable problems etc) - historical or policital questions I wanted to include "software development tasks", but it would be a distraction. Maybe there will be a good benchmark for this, I'm aware there are plenty already. Maybe AI will be capable to be a better software developer than me in some capacity, so I don't want to include this part here. That also maps pretty well to "the better the problem description, the better the output", which doesn't seem to work so neatly with the other categories of tasks and questions. Even if the whole body of questions/tasks/prompts would be very constrained and cover only a single domain, it seems impossible to guarantee that such benchmark is "bias-free" (I know AGI folks love this word). Maybe in some interesting special cases? For example, very constrained and clearly defined classes of questions, at which point, the "language" part of LLMs seems to become less important and more of a distraction. Sure, AI is not just LLMs, and LLMs are not just assistants, and Neural Networks are not just LLMs... There the problem begins to be honest: I don't even know how to align the "benchmark" claims with the kind of AI they are examinin and the ones I know exist. Sure it's possible to benchmark how well an AI decides whether, for example, a picture shows a rabbit. Even then: for some pictures, it's gotta be undecidable, no matter how good the training data is? I'm just a complete layman and commenting about this; I'm not even fluent in the absolute basics of artificial neural networks like perceptrons, gradient descent, backpropagation and typical non-LLM CNNs that are used today, GANs etc. I am and was impressed by AI and deep learning, but to this day I am thorougly disappointed by the hubris of snakeoil salespeople who think it's valuable and meaningful to "benchmark" machines on "general reasoning". I mean, it's already a thing in humans. There are IQ tests for the non-trivia parts. And even these have plenty of discussion revolving around them, for good reason. Is there some "AI benchmark" that exclusively focuses on doing recent IQ tests on models, preferably editions that were published after the particular knowledge cutoff of the respective models? I found (for example) this study [1], but to be honest, I'm not the kind of person who is able to get the core insights presented in such a paper by skimming through it. Because I think there are impressive results, it's just becomimg very hard to see through the bullshit at as an average person. I would also love to understand mroe about the current state of the research on the "LLMs as compression" topic [2][3]. [1] https://arxiv.org/pdf/2507.20208 https://arxiv.org/pdf/2507.20208 [2] https://www.mattmahoney.net/dc/text.html https://www.mattmahoney.net/dc/text.html [3] https://arxiv.org/abs/2410.21352 https://arxiv.org/abs/2410.21352
SurceBeats 1y ago
Benchmarks optimize for fundraising, not users. The gap between "state of the art" and "previous gen" keeps shrinking in real-world use, but investors still write checks based on decimal points in test scores.
- yawnxyz 1y ago
  we try to make benchmarks for users, but it's like that 20% article - different people want different 20% and you just end up adding "features" and whackamoling the different kinds of 20% if a single benchmark could be a universal truth, and it was easy to figure out how to do it, everyone would love that.. but that's why we're in the state we're in right now
  - DrewADesign 1y ago
    The problem isn’t with the benchmarks (or the models, for that matter) it’s their being used to prop up the indefensible product marketing claims made by people frantically justifying asking for more dump trucks of thousand-dollar bills to replace the ones they just burned through in a few months.
    - yawnxyz 1y ago
      unfortunately as benchmark makers we can't really do anything about human nature :shrug:
      - DrewADesign 1y ago
        Absolutely not. This is not a problem with any part of the engineering process. Nearly everything wrong with the AI business lies at the feet of product managers, marketing, the c-suite crowd, etc.
RA_Fisher 1y ago
For statistical AI models, we can use out of sample prediction error as an objective measure to compare models. What makes evaluating LLMs difficult is that comparisons are inextricable from utility (whereas statistical AI models do have a pre-utility step wherein it can be shown out of sample prediction epsilon is minimized).
SkyPuncher 1y ago
Benchmarks are nothing more than highly contextual specs (in traditional code). They demonstrate your code works in a certain way in certain use cases, but they do not prove your code works as expected in all use cases.
- embedding-shape 1y ago
  > Program testing can be used to show the presence of bugs, but never to show their absence. Edsger W. Dijkstra Maybe we need something similar for benchmarks, and updated for today's LLMs, like: > LLM benchmarks can be used to show what tasks they can do, but never to show what tasks they cannot.
pahae 1y ago
I wish the big providers would offer some sort of trial period where you can evaluate models in a _realistic_ setting yourself (i.e cli tools or IDE integrations). I wouldn't even mind strict limits -- just give me two hours or so of usage and I'd already be happy. Seriously. My use-case is probably pretty far from the usual tasks: I'm currently implementing a full observability platform based on VictoriaMetrics / Victorialogs + Grafana. It's quite elaborate and has practically no overlap with the usual/cloud solutions you find out there. For example, it uses an authenticated query stack: I use the Grafana oauth token to authenticate queries by injecting matchers via prom-label-proxy and forward that to promxy for fan-out to different datasources (using the label filter to only query some datasources). The IaC stuff is also not mainstream as I'm not using any of the big cloud providers, but the provider I use nonetheless has a terraform provider. As you can imagine, there's probably not much training data for most of this, so quality of the responses varies widely. From my experience so far Claude (Sonnet 4.5 ) does a _much_ better job than GTP-5 (Codex or normal) with the day-to-day task. Stuff like keeping documentation up to date, spotting inconsistencies, helping me find blind spots in the Alerting rules, etc. It also seems to do better working with provided documentation / links. I've been using Claude for a couple of weeks now but recently switched to codex after my subscription to Claude ran out. I was really curious after reading a lot of good things about it but I gotta say, so far, I'm not impressed. Compared to Claude it gives wrong answers much more frequently (at least in this domain). The results it produces take much more effort to clean up than Claude's. Probably on a level where I could just invest the time myself. Might be that I do not yet know how to correctly prompt GPT but giving both tools the same prompt, Claude does a better job 90% of the time. Anyway, I guess this is my long-winded way of saying that the quality of responses "off the beaten track" varies widely and is worth testing several models with. Especially if your work is not 70+% of coding. Even then I guess that many benchmarks have seized being useful by now?
- deleted 1y ago
  [deleted]
- cavisne 1y ago
  https://wuu73.org/blog/aiguide1.html https://wuu73.org/blog/aiguide1.html You can get a lot of free usage out of the models.
bbor 1y ago
I'm already quite put off by the title (it's science -- if you have a better benchmark, publish it!), but the contents aren't great either. It keeps citing numbers about "445 LLM benchmarks" without confirming whether any of the ones they deem insufficiently statistical are used by any of the major players. I've seen a lot of benchmarks, but maybe 20 are used regularly by large labs, max. "For example, if a benchmark reuses questions from a calculator-free exam such as AIME," the study says, "numbers in each problem will have been chosen to facilitate basic arithmetic. Testing only on these problems would not predict performance on larger numbers, where LLMs struggle." For a math-based critique, this seems to ignore a glaring problem: is it even possible to randomly sample all natural numbers? As another comment pointed out we wouldn't even want to ("LLMs can't accurately multiply 6-digit numbers" isn't something anyone cares about/expected them to do in the first place), but regardless: this seems like a vacuous critique dressed up in a costume of mathematical rigor. At least some of those who design benchmark tests are aware of these concerns. In related news, at least some scientists studying climate change are aware that their methods are imperfect. More at 11! If anyone doubts my concerns and thinks this article is in good faith, just check out this site's "AI+ML" section: https://www.theregister.com/software/ai_ml/ https://www.theregister.com/software/ai_ml/
- daveguy 1y ago
  The article references this review: https://openreview.net/pdf?id=mdA5lVvNcU https://openreview.net/pdf?id=mdA5lVvNcU And the review is pretty damning regarding statistical validity of LLM benchmarks.
- dang 1y ago
  (We've since changed both title and URL - see https://news.ycombinator.com/item?id=45860056 https://news.ycombinator.com/item?id=45860056)
wolttam 1y ago
I'd like to see some video generation benchmarks. For example, one that tested a model's ability to generate POV footage of a humanoid form carrying out typical household tasks Even if it requires human evaluators at first, and even if the models completely suck at this task right now: it seems like the kind of task you'd want them to be good at, if you want these models to eventually carry out these tasks in embodied forms in the real world. Just having the benchmark in the first place is what gives model makers something to optimize for.
- luckydata 1y ago
  Generating footage wouldn't help with the opposite but navigating a simulation would which is a pretty standard type of evaluation for multimodal AIs designed to act in the real world.
  - wolttam 1y ago
    Do you mean that it wouldn't help with ingesting footage and then determining how to act? I can imagine a robotics architecture where you have one model generating footage (next frames for what it is currently seeing) and another dumber model which takes in the generated footage and only knows how to generate the motor/servo control outputs needed to control whatever robot platform it is integrated with. I think that kind of architecture decoupling would be nice. It allows the model with all the world and task-specific knowledge to be agnostic from its underlying robot platform.
lysace 1y ago
Tech companies/bloggers/press/etc are perpetually bad at benchmarks. For browsers they kept pushing simplistic javascript-centric benchmarks even when it was clear for at least 15 years that layout/paint/network/etc were the dominant bottlenecks in real-world usage. It's primarily marketing-driven. I think the technical parts of companies need to attempt to own this more. It gets really weird when engineering priorities shift because of these mostly irrelevant benchmarks.
doctorpangloss 1y ago
The problem with the LLM benchmarks is that if you see one that shows high performance by something that isn’t from Anthropic, Google or OpenAI, you don’t believe it, even if it were “true.” In that sense, benchmarks are a holistic social experience in this domain, less a scientific endeavour.
qustrolabe 1y ago
Technically true but also a very dumb take and manipulative phrasing
riskable 1y ago
We should make a collective git repo full of every kind of annoying bug we (expert developers) can think of. Then use that to benchmark LLMs. Someone want to start? I've got a Yjs/CRDT collaborative editing bug that took like a week and a half of attempts with Claude Code (Sonnet 4.5), GPT5-codex (medium), and GLM-4.6 many, many attempts to figure out. Even then they didn't really get it... Just came up with a successful workaround (which is good enough for me but still...). Aside: You know what really moved the progress bar on finding and fixing the bug? When I had a moment of inspiration and made the frontend send all it's logs to the backend so the AIs could see what was actually happening on the frontend (near real-time). Really, I was just getting sick of manual testing and pasting the console output into the chat (LOL). Laziness FTW! I have the Google Chrome Dev Tools MCP but for some reason it doesn't work as well :shrug:
- simonw 1y ago
  Have you tried the Playwright libraries? Not the MCP, instead telling Claude Code to use the Node.js or Python Playwright libraries directly. I have had some really good results for this for gnarly frontend challenges.
  - kvirani 1y ago
    Curious why not the MCP? I use that
    - s900mhz 1y ago
      When I have a bug I’m iterating on it’s much easier and faster to have it write out the playwright script. That way it does not have to waste time or tokens performing the same actions over and over again. Think of it as TDD.
    - simonw 1y ago
      I don't really like MCPs, at least when I'm working with coding agents like Claude Code or Codex CLI. I'd rather let the agents write code that can do anything the underlying library is capable of, rather than restricting them to just the functionality that the MCP exposes. It's more token efficient too since I don't need to load the full MCP description into my context.
jstummbillig 1y ago
Benchmarks are like SAT scores. Can they guarantee you'll be great at your future job? No, but we are still roughly okay with what they signify. Clearly LLMs are getting better in meaningful ways, and benchmarks correlate with that to some extend.
- zeroonetwothree 1y ago
  There’s no a priori reason to expect a test designed to test human academic performance would be a good one to test LLM job performance. For example a test of “multiply 1765x9392” would have some correlation with human intelligence but it wouldn’t make sense to apply it to computers.
  - sroussey 1y ago
    Actually… ask gpt1 to multiply 1765x9392.
    - SV_BubbleTime 1y ago
      I wish this was more broadly, explained to people… There are LLMs, the engines that make these products run, and then the products themselves. GPT anything should not be asked math problems. LLMs are language models, not math. The line is going to get very blurry because ChatGPT, or Claude or Gemini, are not LLM’s. Their products driven by LLMs. The question or requisite should not be can my LLM do math. It can I build a product that is LLM driven that can reason through math problems. Those are different things. A coworker of mine told me that GPT’s LLM can use Excel files. No, it can’t. But the tools they plugged into it can.
      - deleted 1y ago
        [deleted]
      - lossyalgo 1y ago
        > A coworker of mine told me that GPT’s LLM can use Excel files. No, it can’t. But the tools they plugged into it can. And there's a 50/50 chance they'll use the right tool for the job. I tried the math question above multiple times on gpt5 and it gets it right about 50% of the time. If i ask to "try again" it usually gets it on the 2nd or 3rd try. Most times that it's wrong, it's not far off but it looks deceptively accurate at first glance.
AbrahamParangi 1y ago
A test doesn't need to be objectively meaningful or rigorous in any sense in order to still be useful for comparative ranking.
- hobs 1y ago
  yes it does - it has to be meaningful or rigorous for the comparative ranking to be meaningful or rigorous, or else wtf are you doing? Say I have all the information on my side but only these questions that you are showing the user? Who cares about that comparison?
  - deleted 1y ago
    [deleted]
  - AbrahamParangi 1y ago
    objectively vs comparatively
instagraham 1y ago
I've written about Humanity's Last Exam, which crowdsources tough questions for AI models from domain experts around the world. https://www.happiesthealth.com/articles/future-of-health/humanitys-last-exam-and-the-indian-researchers-testing-ais-limits https://www.happiesthealth.com/articles/future-of-health/hum... It's a shifting goalpost, but one of the things that struck me was how some questions could still be trivial for a fairly qualified human (a doctor in this case) but difficult for an AI model. Reasoning, visual or logic, is built on a set of assumptions that are better gained through IRL experience than crawling datasets and matching answers. This leads me to believe that much of the future for training AI models will lie in exposing them to "meatspace" and annotating their inferences, much like how we train a child. This is a long, long process, and one that is already underway at scale. But it's what might give us emergent intelligences rather than just a basket of competing yet somehow-magic thesaurus.
- sroussey 1y ago
  Mercor is doing doing nine digit per year revenue doing just that. Micro1 and others also.
zeroonetwothree 1y ago
Humans are much better at out of sample prediction than LLMs. And inherently benchmarks cannot be out of sample. So I believe that leads to the disconnect between LLMs getting better and better at in sample prediction (benchmarks) while not improving nearly as much at out of sample (actual work).
twilightzone 1y ago
"Measuring money turns out to be easier than measuring intelligence." Don't ever change, El Reg.
dehrmann 1y ago
This might explain the zeitgeist that new models feel same-ish, despite model developers saying they're getting spectacularly better.
inavida 1y ago
They should laugh while they can ;) Still waiting for the crash and to see what lives on and what gets recycled. My bet is that grok is here to stay ;) (Don't hurt me, I just like his chatbot. It's the best I've tried at, "Find the passage in X that reminded me of the passage in Y given this that and the other thing." It has a tendency to blow smoke if you let it, but they all seek to affirm more than I'd like, but ain't that the modern world? It can also be hilariously funny in surprisingly apt ways.)
- typpilol 1y ago
  Grok is terrible at coding though.
  - deleted 1y ago
    [deleted]
  - JumpCrisscross 1y ago
    If models get commoditised, distribution (and vertical integration) become key. OpenAI and xAI are the only companies that seem to be well hedged for this risk.
  - inavida 1y ago
    Heh. I haven't tried it yet, but even grok says Claude is the way to go.
dupdup 1y ago
for me the definition of AGI is the tool to measure https://arxiv.org/html/2510.18212v2 https://arxiv.org/html/2510.18212v2
dang 1y ago
Url changed from https://www.theregister.com/2025/11/07/measuring_ai_models_hampered_by/ https://www.theregister.com/2025/11/07/measuring_ai_models_h..., which points to this.
gradus_ad 1y ago
AI detractors can say whatever. As a developer Claude Code is almost an unfair cheat code. AI valuations may be absurd but the hype is justified.
Havoc 1y ago
Id hope anyone using LLMs in production is testing them against their use directly. Benchmarks make for a good first pass though to figure out which ones to test
proc0 1y ago
This wasn't that hard to see. > Our systematic review of 445 benchmarks reveals prevalent gaps that undermine the construct validity needed to accurately measure targeted phenomena Intelligence has an element of creativity, and as such the true measurement would be on metrics related to novelty, meaning tasks that have very little resemblance to any other existing task. Otherwise it's hard to parse out whether it's solving problems based on pattern recognition instead of actual reasoning and understanding. In other words, "memorizing" 1000 of the same type of problem, and solving #1001 of that type is not as impressive as solving a novel problem that has never been seen before. Of course this presents challenges to creating the tests because you have to avoid however many petabytes of training data these systems are trained with. That's where some of the illusion of intelligence arises from (illusion not because it's artificial, since there's no reason to think the brain algorithms cannot be recreated in software).
- peacebeard 1y ago
  In my opinion a major weakness in how people reason about this issue is that they describe solving problems as EITHER recall of an existing solution OR creative problem solving. Sure it is possible for a specific solution to be recalled, but it's not possible for a problem to be absolutely unrelated to anything the system has ever seen before and still be solvable. There are many shades of gray in the similarity a problem may have to previously seen problems. In fact, I expect that there are as many shades of gray as there are problems.
  - proc0 1y ago
    The difference is that humans don't memorize petabytes of problems, so from a relative perspective people are constantly solving novel problems they never saw before. I'm thinking this is a requirement for dynamic, few-shot learning. We can clearly see LLMs fail when you throw even a small wrench in the prompt.
    - peacebeard 1y ago
      Humans encounter massive numbers of problems in their experience that informs their problem solving. The same is true of LLM. LLMs do not actually have all their training data memorized. I’m not sure what your basis is for saying “LLMs fail if there is a small wrench in the prompt.” They also succeed despite wrenches in the prompt with great regularity.
lielvilla 1y ago
I’m working a lot with TTS (Text-to-Speach), and it’s also a total wild west - even worse than LLMs in some ways. The demos are always perfect, but once you generate hundreds of minutes you start seeing volume drift, pacing changes, random artifacts, and occasional mispronunciations that never show up in the curated clips. The big difference from LLMs is that we don’t really have production-grade, standardized benchmarks for long-form TTS. We need things like volume-stability across segments, speech-rate consistency, and pronunciation accuracy over a hard corpus. I wrote up what this could look like here: https://lielvilla.com/blog/death-of-demo/ https://lielvilla.com/blog/death-of-demo/
naasking 1y ago
Clearly we need tests that check for effectiveness at applying general mathematical, logical and relational operations, eg. set theory, relational algebra, first and second order logic, type theory, the lambda calculus, recurrence and induction, etc., and the ability to use these to abstract over specifics and the ability to generalize. The upside is that these can all be generated and checked synthetically so large data sets are possible, in both formal and natural languages.
rumble_poster 1y ago
On top of this, we have complex infrastructure setups with different TPUs and GPUs across multiple data centers. The benchmarks tested when models were released might not reflect what we're actually using now. We need to evaluate models continuously, for example, https://isitnerfed.org/ https://isitnerfed.org/ does exactly that.

416 points by pseudolus 1y ago

Marshferm 1y ago

Don’t get high on your own supply.

calpaterson 1y ago

Definitely one of the weaker areas in the current LLM boom. Comparing models, or even different versions of the same model, is a pseudo-scientific mess. I'm still using https://lmarena.ai/leaderboard https://lmarena.ai/leaderboard. Perhaps there is something better and someone will pipe up to tell me about it. But we use LLMs at work and have unexplainable variations between them. And when we get a prompt working reliably on one model, we often have trouble porting it to another LLM - even straight "version upgrades" such as from GPT-4 to -5. Your prompt and your model become highly coupled quite easily. I dunno what to do about it and am tending to just pick Gemini as a result.

HPsquared 1y ago
Psychometric testing of humans has a lot of difficulties, too. It's hard to measure some things.
ACCount37 1y ago
Ratings on LMArena are too easily gamed. Even professional human evaluators are quite vulnerable to sycophancy and overconfident-and-wrong answers. And LMArena evaluators aren't professionals. A lot of the sycophancy mess that seeps from this generation of LLM stems from reckless tuning based on human feedback. Tuning for good LMArena performance has similar effects - and not at all by a coincidence.
- energy123 1y ago
  It's biased to small context performance, which is why I don't pay much attention to it as a developer aside from a quick glance. I need performance at 40-100k tokens which models like Deepseek can't deliver but Gemini 2.5 Pro and ChatGPT 5.0 Thinking can.
  - ACCount37 1y ago
    And even "long term performance" splits itself into "performance on multi-turn instruction following" and "performance on agentic tasks" down the line. And "performance on agentic tasks" is a hydra in itself. Capturing LLM performance with a single metric is a hopeless task. But even a single flawed metric beats no metrics at all.
diamond559 1y ago
I'd rather quit then be forced to beta test idiocracy. What's your company so we can all avoid it?
botro 1y ago
This is something I've stuggled with for my site, I made https://aimodelreview.com/ https://aimodelreview.com/ to compare the outputs of LLMs over a variety of prompts and categories, allowing a side by side comparison between them. I ran each prompt 4 times for each model with different temperature values available as a toggles. My thinking was to just make the responses available to users and let them see how models perform. But from some feedback, turns out users don't want to have to evaluate the answers and would rather see a leaderboard and rankings. The scalable solution to that would be LLM as judge that some benchmarks already use, but that just feels wrong to me. LM Arena tries to solve this with the crowd sourced solution, but I think the right method would have to be domain expert human reviewers, so like Wirecutter VS IMDb, but that is expensive to pull off.
3abiton 1y ago
> Comparing models, or even different versions of the same model, is a pseudo-scientific mess. Reminder that in most cases, it's impossible to know if there is cross-contamination from the test set of the public benchmarks, as most LLMs are not truely open-source. We can't replicate them. So arguably it's worse in some cases, pretty much fraud if you account for the VC money pouring in. This is even more evident in unknown models from lesser known institutes like from UAE.
andai 1y ago
>when we get a prompt working reliably on one model, we often have trouble porting it to another LLM I saw a study where a prompt massively boosted one model's performance on a task, but significantly reduced another popular model's performance on the same task.
- boccaff 1y ago
  Do you have any pointer to search for that?

bubblelicious 1y ago

I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster. No one really has a good solution, and researchers are also in a huge rush and don’t want to end up making their whole job benchmarking. Even if you could, and even if you have the right background you can do benchmarks full time and they still would be a mess. Product testing (with traditional A/B tests) are kind of the best bet since you can measure what you care about _directly_ and at scale. I would say there is of course “benchmarketing” but generally people do sincerely want to make good benchmarks it’s just hard or impossible. For many of these problems we’re hitting capabilities where we don’t even have a decent paradigm to use,

ACCount37 1y ago
A/B testing is radioactive too. It's indirectly optimizing for user feedback - less stupid than directly optimizing for user feedback, but still quite dangerous. Human raters are exploitable, and you never know whether the B has a genuine performance advantage over A, or just found a meat exploit by an accident. It's what fucked OpenAI over with 4o, and fucked over many other labs in more subtle ways.
- bubblelicious 1y ago
  Are you talking about just preferences or A/B tests on like retention and engagement? The latter I think is pretty reliable and powerful though I have never personally done them. Preferences are just as big a mess: WHO the annotators are matters, and if you are using preferences as a proxy for like correctness, you’re not really measuring correctness you’re measuring e.g. persuasion. A lot of construct validity challenges (which themselves are hard to even measure in domain).
  - ACCount37 1y ago
    Yes. All of them are poisoned metrics, just in different ways. GPT-4o's endless sycophancy was great for retention, GPT-5's style of ending every response in a question is great for engagement. Are those desirable traits though? Doubt it. They look like simple tricks and reek of reward hacking - and A/B testing rewards them indeed. Direct optimization is even worse. Combining the two is ruinous. Mind, I'm not saying that those metrics are useless. Radioactive materials aren't useless. You just got to keep their unpleasant properties in mind at all times - or suffer the consequences.

jennyholzer 1y ago

I've been getting flagged by high-on-their-own-supply AI boosters for identifying that LLM benchmarks have been obvious bullshit for at least the last year and a half. What changed to make "the inevitable AI bubble" the dominant narrative in last week or so?

conception 1y ago
Companies are talking about needing trillions of dollars is why.
- foobiekr 1y ago
  And the government backstops. Nothing says confidence that AGI is imminent like needing the US government to prevent your investments from losing you money.
HPsquared 1y ago
Benchmarks in general have this problem, across pretty much all industries. "When a measure becomes a target" and all that.
icameron 1y ago
The market was down for AI related stocks especially, while down only over 3% it’s the worst week since April, and there’s no single event that is to blame it just looks like market sentiment has shifted away from the previous unchecked exuberance.
Kiro 1y ago
Link those comments please because I checked your history and the flagged ones were pure nonsense with zero insights. Also, calling out LLM benchmarks has never been a radical take and basically the default on this site.
purple_turtle 1y ago
It is possible to be right on the main theme but only by accident (with arguments and claims being wrong), communicating in a highly faulty way, with pointless insults, doing it in offtopic derails, being correct on minor point while being mostly wrong etc. Can you link some of these comments you consider useful but got flagged?

mikert89 1y ago

[flagged]

jennyholzer 1y ago
[flagged]
ZyanWu 1y ago
Any argument to back up your claim?
- HPsquared 1y ago
  The fact that I could type this exchange into an LLM and get a list of reasonable arguments.
- CooCooCaCha 1y ago
  Not the person you replied to, but I'm a programmer and up until ~3 weeks ago I really only used AI for auto-complete, looking up API information, and constructing arcane CLI commands. I decided to take a leap and use AI as much as possible to complete a ticket at work. Now, 3 weeks later AI is writing 90% of my code. Granted, I'm not sitting back sipping on a latte while AI does my job. It's a very interactive process, I spend more time reviewing code and going back and forth with the AI to get the result I want. But it's become surprisingly good. I wouldn't say I'm anywhere close to 10x more productive, but perhaps 50%.
  - GPerson 1y ago
    It definitely has its amazing moments, but sometimes I get caught in a loop of expecting it to do the thing, it not working, and spinning my wheels a lot instead of just solving it myself. I think I’m still learning how to use the tools effectively, but the random nature of it makes it difficult.
  - diamond559 1y ago
    Yeah, nobody believes your made up numbers buddy. You're a stonk pumper, we know it.

shanev 1y ago

This is solvable at the level of an individual developer. Write your own benchmark for code problems that you've solved. Verify tests pass and that it satisfies your metrics like tok/s and TTFT. Create a harness that works with API keys or local models (if you're going that route).

motoboi 1y ago
Well, openai github is open to write evaluations. Just add your there and guaranteed that the next model will perform better on them.
cactusplant7374 1y ago
I think that's what this site is doing: https://aistupidlevel.info/ https://aistupidlevel.info/
davedx 1y ago
That’s called evals and yes any serious AI project uses them
hamdingers 1y ago
At the developer level all my LLM use is in the context of agentic wrappers, so my benchmark is fairly trivial: Configure aider or claude code to use the new model, try to do some work. The benchmark is pass/fail, if after a little while I feel the performance is better than the last model I was using it's a pass, otherwise it's a fail and I go back. Building your own evaluations makes sense if you're serving an LLM up to customers and want to know how it performs, but if you are the user... use it and see how it goes. It's all subjective anyway.
- embedding-shape 1y ago
  > Building your own evaluations makes sense if you're serving an LLM up to customers and want to know how it performs, but if you are the user... use it and see how it goes. It's all subjective anyway. I'd really caution against this approach, mainly because humans suck at removing emotions and other "human" factors when judging how well something works, but also because comparing across models gets a lot easier when you can see 77/100 vs 91/100 as a percentage score, over your own tasks that you actually use the LLMs for. Just don't share this benchmark publicly once you're using it for measurements.

bee_rider 1y ago

> "For example, if a benchmark reuses questions from a calculator-free exam such as AIME," the study says, "numbers in each problem will have been chosen to facilitate basic arithmetic. Testing only on these problems would not predict performance on larger numbers, where LLMs struggle." When models figure out how to exploit an effect that every clever college student does, that should count as a win. That’s a much more human-like reasoning ability, than the ability to multiply large numbers or whatever (computers were already good at that, to the point that it has become a useless skill for humans to have). The point of these LLMs is to do things that computers were bad at.

layer8 1y ago
I don’t think the fact that LLMs can handle small numbers more reliably has anything to do with their reasoning ability. To the contrary, reasoning ability should enable them to handle numbers of arbitrary size, just as it enables humans to do so, given some pencil and paper. However: > Testing only on these problems would not predict performance on larger numbers, where LLMs struggle. Since performance on large numbers is not what these exams are intended to test for, I don’t see this as a counterargument, unless the benchmarks are misrepresenting what is being tested for.
- zamadatix 1y ago
  Pencil and paper is just testing with tools enabled.
  - layer8 1y ago
    You seem to be addressing an argument that wasn’t made. Personally, I’d say that such tool use is more akin to a human using a calculator.
    - zamadatix 1y ago
      I'm not addressing an argument, just stating that's already a form of LLM testing done today for people wanting to look at the difference in results the same as the human analogy.

moritzwarhier 1y ago

When people claim that there is such a thing as "X% accuracy in reasoning", it's really hard to take anything else seriously, no matter how impressive. AI (and humans!) aside, claiming that there was an oracle that could "answer all questions" is a solved problem. Such a thing cannot exist. But this is going already too deep IMO. When people start talking about percentages or benchmark scores, there has to be some denominator. And there can be no bias-free such denominator for - trivia questions - mathematical questions (oh, maybe I'm wrong here, intuitively I'd say it's impossible for various reasons: varying "hardness", undecidable problems etc) - historical or policital questions I wanted to include "software development tasks", but it would be a distraction. Maybe there will be a good benchmark for this, I'm aware there are plenty already. Maybe AI will be capable to be a better software developer than me in some capacity, so I don't want to include this part here. That also maps pretty well to "the better the problem description, the better the output", which doesn't seem to work so neatly with the other categories of tasks and questions. Even if the whole body of questions/tasks/prompts would be very constrained and cover only a single domain, it seems impossible to guarantee that such benchmark is "bias-free" (I know AGI folks love this word). Maybe in some interesting special cases? For example, very constrained and clearly defined classes of questions, at which point, the "language" part of LLMs seems to become less important and more of a distraction. Sure, AI is not just LLMs, and LLMs are not just assistants, and Neural Networks are not just LLMs... There the problem begins to be honest: I don't even know how to align the "benchmark" claims with the kind of AI they are examinin and the ones I know exist. Sure it's possible to benchmark how well an AI decides whether, for example, a picture shows a rabbit. Even then: for some pictures, it's gotta be undecidable, no matter how good the training data is? I'm just a complete layman and commenting about this; I'm not even fluent in the absolute basics of artificial neural networks like perceptrons, gradient descent, backpropagation and typical non-LLM CNNs that are used today, GANs etc. I am and was impressed by AI and deep learning, but to this day I am thorougly disappointed by the hubris of snakeoil salespeople who think it's valuable and meaningful to "benchmark" machines on "general reasoning". I mean, it's already a thing in humans. There are IQ tests for the non-trivia parts. And even these have plenty of discussion revolving around them, for good reason. Is there some "AI benchmark" that exclusively focuses on doing recent IQ tests on models, preferably editions that were published after the particular knowledge cutoff of the respective models? I found (for example) this study [1], but to be honest, I'm not the kind of person who is able to get the core insights presented in such a paper by skimming through it. Because I think there are impressive results, it's just becomimg very hard to see through the bullshit at as an average person. I would also love to understand mroe about the current state of the research on the "LLMs as compression" topic [2][3]. [1] https://arxiv.org/pdf/2507.20208 https://arxiv.org/pdf/2507.20208 [2] https://www.mattmahoney.net/dc/text.html https://www.mattmahoney.net/dc/text.html [3] https://arxiv.org/abs/2410.21352 https://arxiv.org/abs/2410.21352

SurceBeats 1y ago

Benchmarks optimize for fundraising, not users. The gap between "state of the art" and "previous gen" keeps shrinking in real-world use, but investors still write checks based on decimal points in test scores.

yawnxyz 1y ago
we try to make benchmarks for users, but it's like that 20% article - different people want different 20% and you just end up adding "features" and whackamoling the different kinds of 20% if a single benchmark could be a universal truth, and it was easy to figure out how to do it, everyone would love that.. but that's why we're in the state we're in right now
- DrewADesign 1y ago
  The problem isn’t with the benchmarks (or the models, for that matter) it’s their being used to prop up the indefensible product marketing claims made by people frantically justifying asking for more dump trucks of thousand-dollar bills to replace the ones they just burned through in a few months.
  - yawnxyz 1y ago
    unfortunately as benchmark makers we can't really do anything about human nature :shrug:
    - DrewADesign 1y ago
      Absolutely not. This is not a problem with any part of the engineering process. Nearly everything wrong with the AI business lies at the feet of product managers, marketing, the c-suite crowd, etc.

RA_Fisher 1y ago

For statistical AI models, we can use out of sample prediction error as an objective measure to compare models. What makes evaluating LLMs difficult is that comparisons are inextricable from utility (whereas statistical AI models do have a pre-utility step wherein it can be shown out of sample prediction epsilon is minimized).

SkyPuncher 1y ago

Benchmarks are nothing more than highly contextual specs (in traditional code). They demonstrate your code works in a certain way in certain use cases, but they do not prove your code works as expected in all use cases.

embedding-shape 1y ago
> Program testing can be used to show the presence of bugs, but never to show their absence. Edsger W. Dijkstra Maybe we need something similar for benchmarks, and updated for today's LLMs, like: > LLM benchmarks can be used to show what tasks they can do, but never to show what tasks they cannot.

pahae 1y ago

I wish the big providers would offer some sort of trial period where you can evaluate models in a _realistic_ setting yourself (i.e cli tools or IDE integrations). I wouldn't even mind strict limits -- just give me two hours or so of usage and I'd already be happy. Seriously. My use-case is probably pretty far from the usual tasks: I'm currently implementing a full observability platform based on VictoriaMetrics / Victorialogs + Grafana. It's quite elaborate and has practically no overlap with the usual/cloud solutions you find out there. For example, it uses an authenticated query stack: I use the Grafana oauth token to authenticate queries by injecting matchers via prom-label-proxy and forward that to promxy for fan-out to different datasources (using the label filter to only query some datasources). The IaC stuff is also not mainstream as I'm not using any of the big cloud providers, but the provider I use nonetheless has a terraform provider. As you can imagine, there's probably not much training data for most of this, so quality of the responses varies widely. From my experience so far Claude (Sonnet 4.5 ) does a _much_ better job than GTP-5 (Codex or normal) with the day-to-day task. Stuff like keeping documentation up to date, spotting inconsistencies, helping me find blind spots in the Alerting rules, etc. It also seems to do better working with provided documentation / links. I've been using Claude for a couple of weeks now but recently switched to codex after my subscription to Claude ran out. I was really curious after reading a lot of good things about it but I gotta say, so far, I'm not impressed. Compared to Claude it gives wrong answers much more frequently (at least in this domain). The results it produces take much more effort to clean up than Claude's. Probably on a level where I could just invest the time myself. Might be that I do not yet know how to correctly prompt GPT but giving both tools the same prompt, Claude does a better job 90% of the time. Anyway, I guess this is my long-winded way of saying that the quality of responses "off the beaten track" varies widely and is worth testing several models with. Especially if your work is not 70+% of coding. Even then I guess that many benchmarks have seized being useful by now?

deleted 1y ago
[deleted]
cavisne 1y ago
https://wuu73.org/blog/aiguide1.html https://wuu73.org/blog/aiguide1.html You can get a lot of free usage out of the models.

bbor 1y ago

I'm already quite put off by the title (it's science -- if you have a better benchmark, publish it!), but the contents aren't great either. It keeps citing numbers about "445 LLM benchmarks" without confirming whether any of the ones they deem insufficiently statistical are used by any of the major players. I've seen a lot of benchmarks, but maybe 20 are used regularly by large labs, max. "For example, if a benchmark reuses questions from a calculator-free exam such as AIME," the study says, "numbers in each problem will have been chosen to facilitate basic arithmetic. Testing only on these problems would not predict performance on larger numbers, where LLMs struggle." For a math-based critique, this seems to ignore a glaring problem: is it even possible to randomly sample all natural numbers? As another comment pointed out we wouldn't even want to ("LLMs can't accurately multiply 6-digit numbers" isn't something anyone cares about/expected them to do in the first place), but regardless: this seems like a vacuous critique dressed up in a costume of mathematical rigor. At least some of those who design benchmark tests are aware of these concerns. In related news, at least some scientists studying climate change are aware that their methods are imperfect. More at 11! If anyone doubts my concerns and thinks this article is in good faith, just check out this site's "AI+ML" section: https://www.theregister.com/software/ai_ml/ https://www.theregister.com/software/ai_ml/

daveguy 1y ago
The article references this review: https://openreview.net/pdf?id=mdA5lVvNcU https://openreview.net/pdf?id=mdA5lVvNcU And the review is pretty damning regarding statistical validity of LLM benchmarks.
dang 1y ago
(We've since changed both title and URL - see https://news.ycombinator.com/item?id=45860056 https://news.ycombinator.com/item?id=45860056)

wolttam 1y ago

I'd like to see some video generation benchmarks. For example, one that tested a model's ability to generate POV footage of a humanoid form carrying out typical household tasks Even if it requires human evaluators at first, and even if the models completely suck at this task right now: it seems like the kind of task you'd want them to be good at, if you want these models to eventually carry out these tasks in embodied forms in the real world. Just having the benchmark in the first place is what gives model makers something to optimize for.

luckydata 1y ago
Generating footage wouldn't help with the opposite but navigating a simulation would which is a pretty standard type of evaluation for multimodal AIs designed to act in the real world.
- wolttam 1y ago
  Do you mean that it wouldn't help with ingesting footage and then determining how to act? I can imagine a robotics architecture where you have one model generating footage (next frames for what it is currently seeing) and another dumber model which takes in the generated footage and only knows how to generate the motor/servo control outputs needed to control whatever robot platform it is integrated with. I think that kind of architecture decoupling would be nice. It allows the model with all the world and task-specific knowledge to be agnostic from its underlying robot platform.

lysace 1y ago

Tech companies/bloggers/press/etc are perpetually bad at benchmarks. For browsers they kept pushing simplistic javascript-centric benchmarks even when it was clear for at least 15 years that layout/paint/network/etc were the dominant bottlenecks in real-world usage. It's primarily marketing-driven. I think the technical parts of companies need to attempt to own this more. It gets really weird when engineering priorities shift because of these mostly irrelevant benchmarks.

doctorpangloss 1y ago

The problem with the LLM benchmarks is that if you see one that shows high performance by something that isn’t from Anthropic, Google or OpenAI, you don’t believe it, even if it were “true.” In that sense, benchmarks are a holistic social experience in this domain, less a scientific endeavour.

qustrolabe 1y ago

Technically true but also a very dumb take and manipulative phrasing

riskable 1y ago

We should make a collective git repo full of every kind of annoying bug we (expert developers) can think of. Then use that to benchmark LLMs. Someone want to start? I've got a Yjs/CRDT collaborative editing bug that took like a week and a half of attempts with Claude Code (Sonnet 4.5), GPT5-codex (medium), and GLM-4.6 many, many attempts to figure out. Even then they didn't really get it... Just came up with a successful workaround (which is good enough for me but still...). Aside: You know what really moved the progress bar on finding and fixing the bug? When I had a moment of inspiration and made the frontend send all it's logs to the backend so the AIs could see what was actually happening on the frontend (near real-time). Really, I was just getting sick of manual testing and pasting the console output into the chat (LOL). Laziness FTW! I have the Google Chrome Dev Tools MCP but for some reason it doesn't work as well :shrug:

simonw 1y ago
Have you tried the Playwright libraries? Not the MCP, instead telling Claude Code to use the Node.js or Python Playwright libraries directly. I have had some really good results for this for gnarly frontend challenges.
- kvirani 1y ago
  Curious why not the MCP? I use that
  - s900mhz 1y ago
    When I have a bug I’m iterating on it’s much easier and faster to have it write out the playwright script. That way it does not have to waste time or tokens performing the same actions over and over again. Think of it as TDD.
  - simonw 1y ago
    I don't really like MCPs, at least when I'm working with coding agents like Claude Code or Codex CLI. I'd rather let the agents write code that can do anything the underlying library is capable of, rather than restricting them to just the functionality that the MCP exposes. It's more token efficient too since I don't need to load the full MCP description into my context.

jstummbillig 1y ago

Benchmarks are like SAT scores. Can they guarantee you'll be great at your future job? No, but we are still roughly okay with what they signify. Clearly LLMs are getting better in meaningful ways, and benchmarks correlate with that to some extend.

zeroonetwothree 1y ago
There’s no a priori reason to expect a test designed to test human academic performance would be a good one to test LLM job performance. For example a test of “multiply 1765x9392” would have some correlation with human intelligence but it wouldn’t make sense to apply it to computers.
- sroussey 1y ago
  Actually… ask gpt1 to multiply 1765x9392.
  - SV_BubbleTime 1y ago
    I wish this was more broadly, explained to people… There are LLMs, the engines that make these products run, and then the products themselves. GPT anything should not be asked math problems. LLMs are language models, not math. The line is going to get very blurry because ChatGPT, or Claude or Gemini, are not LLM’s. Their products driven by LLMs. The question or requisite should not be can my LLM do math. It can I build a product that is LLM driven that can reason through math problems. Those are different things. A coworker of mine told me that GPT’s LLM can use Excel files. No, it can’t. But the tools they plugged into it can.
    - deleted 1y ago
      [deleted]
    - lossyalgo 1y ago
      > A coworker of mine told me that GPT’s LLM can use Excel files. No, it can’t. But the tools they plugged into it can. And there's a 50/50 chance they'll use the right tool for the job. I tried the math question above multiple times on gpt5 and it gets it right about 50% of the time. If i ask to "try again" it usually gets it on the 2nd or 3rd try. Most times that it's wrong, it's not far off but it looks deceptively accurate at first glance.

AbrahamParangi 1y ago

A test doesn't need to be objectively meaningful or rigorous in any sense in order to still be useful for comparative ranking.

hobs 1y ago
yes it does - it has to be meaningful or rigorous for the comparative ranking to be meaningful or rigorous, or else wtf are you doing? Say I have all the information on my side but only these questions that you are showing the user? Who cares about that comparison?
- deleted 1y ago
  [deleted]
- AbrahamParangi 1y ago
  objectively vs comparatively

instagraham 1y ago

I've written about Humanity's Last Exam, which crowdsources tough questions for AI models from domain experts around the world. https://www.happiesthealth.com/articles/future-of-health/humanitys-last-exam-and-the-indian-researchers-testing-ais-limits https://www.happiesthealth.com/articles/future-of-health/hum... It's a shifting goalpost, but one of the things that struck me was how some questions could still be trivial for a fairly qualified human (a doctor in this case) but difficult for an AI model. Reasoning, visual or logic, is built on a set of assumptions that are better gained through IRL experience than crawling datasets and matching answers. This leads me to believe that much of the future for training AI models will lie in exposing them to "meatspace" and annotating their inferences, much like how we train a child. This is a long, long process, and one that is already underway at scale. But it's what might give us emergent intelligences rather than just a basket of competing yet somehow-magic thesaurus.

sroussey 1y ago
Mercor is doing doing nine digit per year revenue doing just that. Micro1 and others also.

zeroonetwothree 1y ago

Humans are much better at out of sample prediction than LLMs. And inherently benchmarks cannot be out of sample. So I believe that leads to the disconnect between LLMs getting better and better at in sample prediction (benchmarks) while not improving nearly as much at out of sample (actual work).

twilightzone 1y ago

"Measuring money turns out to be easier than measuring intelligence." Don't ever change, El Reg.

dehrmann 1y ago

This might explain the zeitgeist that new models feel same-ish, despite model developers saying they're getting spectacularly better.

inavida 1y ago

They should laugh while they can ;) Still waiting for the crash and to see what lives on and what gets recycled. My bet is that grok is here to stay ;) (Don't hurt me, I just like his chatbot. It's the best I've tried at, "Find the passage in X that reminded me of the passage in Y given this that and the other thing." It has a tendency to blow smoke if you let it, but they all seek to affirm more than I'd like, but ain't that the modern world? It can also be hilariously funny in surprisingly apt ways.)

typpilol 1y ago
Grok is terrible at coding though.
- deleted 1y ago
  [deleted]
- JumpCrisscross 1y ago
  If models get commoditised, distribution (and vertical integration) become key. OpenAI and xAI are the only companies that seem to be well hedged for this risk.
- inavida 1y ago
  Heh. I haven't tried it yet, but even grok says Claude is the way to go.

dupdup 1y ago

for me the definition of AGI is the tool to measure https://arxiv.org/html/2510.18212v2 https://arxiv.org/html/2510.18212v2

dang 1y ago

Url changed from https://www.theregister.com/2025/11/07/measuring_ai_models_hampered_by/ https://www.theregister.com/2025/11/07/measuring_ai_models_h..., which points to this.

gradus_ad 1y ago

AI detractors can say whatever. As a developer Claude Code is almost an unfair cheat code. AI valuations may be absurd but the hype is justified.

Havoc 1y ago

Id hope anyone using LLMs in production is testing them against their use directly. Benchmarks make for a good first pass though to figure out which ones to test

proc0 1y ago

This wasn't that hard to see. > Our systematic review of 445 benchmarks reveals prevalent gaps that undermine the construct validity needed to accurately measure targeted phenomena Intelligence has an element of creativity, and as such the true measurement would be on metrics related to novelty, meaning tasks that have very little resemblance to any other existing task. Otherwise it's hard to parse out whether it's solving problems based on pattern recognition instead of actual reasoning and understanding. In other words, "memorizing" 1000 of the same type of problem, and solving #1001 of that type is not as impressive as solving a novel problem that has never been seen before. Of course this presents challenges to creating the tests because you have to avoid however many petabytes of training data these systems are trained with. That's where some of the illusion of intelligence arises from (illusion not because it's artificial, since there's no reason to think the brain algorithms cannot be recreated in software).

peacebeard 1y ago
In my opinion a major weakness in how people reason about this issue is that they describe solving problems as EITHER recall of an existing solution OR creative problem solving. Sure it is possible for a specific solution to be recalled, but it's not possible for a problem to be absolutely unrelated to anything the system has ever seen before and still be solvable. There are many shades of gray in the similarity a problem may have to previously seen problems. In fact, I expect that there are as many shades of gray as there are problems.
- proc0 1y ago
  The difference is that humans don't memorize petabytes of problems, so from a relative perspective people are constantly solving novel problems they never saw before. I'm thinking this is a requirement for dynamic, few-shot learning. We can clearly see LLMs fail when you throw even a small wrench in the prompt.
  - peacebeard 1y ago
    Humans encounter massive numbers of problems in their experience that informs their problem solving. The same is true of LLM. LLMs do not actually have all their training data memorized. I’m not sure what your basis is for saying “LLMs fail if there is a small wrench in the prompt.” They also succeed despite wrenches in the prompt with great regularity.

lielvilla 1y ago

I’m working a lot with TTS (Text-to-Speach), and it’s also a total wild west - even worse than LLMs in some ways. The demos are always perfect, but once you generate hundreds of minutes you start seeing volume drift, pacing changes, random artifacts, and occasional mispronunciations that never show up in the curated clips. The big difference from LLMs is that we don’t really have production-grade, standardized benchmarks for long-form TTS. We need things like volume-stability across segments, speech-rate consistency, and pronunciation accuracy over a hard corpus. I wrote up what this could look like here: https://lielvilla.com/blog/death-of-demo/ https://lielvilla.com/blog/death-of-demo/

naasking 1y ago

Clearly we need tests that check for effectiveness at applying general mathematical, logical and relational operations, eg. set theory, relational algebra, first and second order logic, type theory, the lambda calculus, recurrence and induction, etc., and the ability to use these to abstract over specifics and the ability to generalize. The upside is that these can all be generated and checked synthetically so large data sets are possible, in both formal and natural languages.

rumble_poster 1y ago

On top of this, we have complex infrastructure setups with different TPUs and GPUs across multiple data centers. The benchmarks tested when models were released might not reflect what we're actually using now. We need to evaluate models continuously, for example, https://isitnerfed.org/ https://isitnerfed.org/ does exactly that.