4 ms·
These posts are going to be a constant for the next year, because there's no objective way to compare models (past low-level numbers like token generation speed
by 0xbadcafebee 6mo ago
These posts are going to be a constant for the next year, because there's no objective way to compare models (past low-level numbers like token generation speed, average reasoning token amount, # of parameters, active experts, etc). They're all quite different in a lot of ways, they're used for many different things by different people, and they're not deterministic. So you're constantly gonna see benchmarks and tests and proclamations of "THIS model beat THAT model!", with people racing around trying to find the best one.
But there is no best one. There's just the best one for you, based on whatever your criteria is. It's likely we'll end up in a "Windows vs MacOS vs Linux" style world, where people stick to their camps that do a particular thing a particular way.
- verve_rat 6mo agoMy theory is we will end up in a similar spot to hiring people. You can look at a CV (benchmarks) but you won't know for sure until you've worked with them for six months. We as an industry cannot determine if one software engineer is objectively better than another, on practically any dimension, so why do we think we can come to an objective ranking of models?
- zelphirkalt 6mo agoNot many things are as manifold broken as hiring these days. I hope we do not end up there.
- deleted 6mo ago[deleted]
- pishpash 6mo agoYou do not interview 1000 rounds on problems you're actually solving. If you did, hiring would be fine. Minus the social fit aspect, which isn't as relevant for a model.
- roymain 6mo agoThe CV-to-six-months analogy is actually exactly right and it's also why benchmarks for hiring people stopped being useful. The signal that holds up is what you see when something breaks, which is hard to compress into a number.
- bartekpacia 6mo agothis smells like an ai-generated comment so much
- tlb 6mo agoYes, the entire field of software engineering ran aground on not being able to test how well people can write software. But I'm more optimistic about testing programming models. You can run repeated tests, and compare median performance. You can run long tests, like hundreds of hours, while getting more than a few humans to complete half-day tests is a huge project. And you can do ablation testing, where you remove some feature of the environment or tools and see how much it helps/hurts.
- PunchyHamster 6mo agoTerrible comparison. CV is just a list, telling you barely anything about performance and that's when candidate is not lying to get thru HR filter. And we can judge developer performance, it just takes 6 months to a year working with a team so it's just hard to get metric
- ljlolel 6mo ago[flagged]
- idonotknowwhy 6mo agoSo like Open Router?
- ljlolel 6mo agoA secure and open source Open Router
- yorwba 6mo agoThere are objective ways to compare models. They involve repeated sampling and statistical analysis to determine whether the results are likely to hold up in the future or whether they're just a fluke. If you fine-tune each model to achieve its full potential on the task you expect to be giving it, the rankings produced by different benchmarks even agree to a high degree: https://arxiv.org/abs/2507.05195 https://arxiv.org/abs/2507.05195 The author didn't do any of that. They ran each model once on each of 13 (so far) problems and then they chose to highlight the results for the 12th problem. That's not even p-hacking, because they didn't stop to think about p-values in the first place. LLM quality is highly variable across runs, so running each model once tells you about as much about which one is better as flipping two coins once and having one come up heads and the other tails tells you about whether one of them is more biased than the other.
- jiggunjer 6mo agoThat's objective metrics. Not an objective way to compare, which is the selection of metrics to include.
- cromka 6mo agoThat's exactly why there's a ton of different benchmarking suites used for evaluating hardware performance. I reckon we'll have similar suites comparing different aspects of models. And, at some point, we'll be dealing with models skewing results whenever they detect they're being benchmarked, like it happened before with hardware. Some say that's already happening with the pelican test.
- PunchyHamster 6mo ago> I reckon we'll have similar suites comparing different aspects of models. The problem is that hardware benchmarks are harder to game. Yes, hardware manufacturer can make driver tweaks for say particular game to run better but the benchmark is still representable for the workflow user faces and they can't change the most important part, hardware, they can't benchmark gimmick their way in designing hardware Meanwhile in LLM land the game is to tune it for the current popular set of benchmarks, all while user experience is only vaguely related to those results
- pjerem 6mo agoThe news is not in the way to compare models, it’s that Kimi K2.6 (and I’d add Deepseek v4 Pro) are more or less equivalent to Opus and that’s already pretty big. They are open source and cost waaaay less per token than American models. I’m using them right now on the $20 Ollama cloud plan and I can actually work with them on my side projects without reaching the limits too much. With Claude Pro $20 plan my usage can barely survive one or two prompts. And I choose Ollama cloud just because their CLI is convenient to use but their are a lot of other providers for those models so you aren’t even stuck with shitty conditions and usage rules. To me that’s a pretty bad thing for American economy.
- chvid 6mo agoOr maybe it is a pretty good thing for the American economy that you can get AI at cost rather than monopoly pricing. You know, for the rest of the economy that is not big tech.
- deleted 6mo ago[deleted]
- PunchyHamster 6mo agoIt's not good for current administration. The American AI growth is only thing that keeps the GDP not looking terrible. And investor pumping money in US AI circular money flow just makes innovation everywhere else slower. If not for the GPU/Memory drought running stuff locally (or just in competition cloud) would be far cheaper
- arvid-lind 6mo ago> It's not good for current administration I don't know where to begin if you're leading with that. Anything approaching reality is not good for the current administration.
- deleted 6mo ago[deleted]
- 6mo ago
- chrisandchris 6mo agoThat was my thought too. > The Word Gem Puzzle is a sliding-tile letter puzzle. The board is a rectangular grid (10×10, 15×15, 20×20, 25×25, or 30×30) filled with letter tiles and one blank space. Just last week my superior asked to implement that for a customer. /s Maybe some real, real task would be good? Add sone database, some REST, some random JS framework and let it figure out a full-stack task instead of creating some rectangles?
- PunchyHamster 6mo agogiving real relatable task like that is memory excercise, not any reasoning excercise. The training dataset have tens of thousands apps like that
- charcircuit 6mo agoA pretty simple one would be to have every model try and one shot every ticket your company has and then measure the acceptance rate of each model.
- sam_goody 6mo agoExcept that if you tried one-shotting your ticket twenty times at different hours of the day and different days of the week, you would have enough changes to make benchmarks even if you used the same model every time. Much moreso if you fiddled with the thinking or changed the prompt. Because non-deterministic, because of constant updates and changes, and because the models are throttled according to number of users, releases, et al.
- serial_dev 6mo agoYou never get "the same" Steph Curry, he might be tired, annoyed by a fan, getting older... but if he and I were to throw 100 3-pointers, we could all correctly guess who will perform better.
- sam_goody 6mo agoGood point. But I use Codex and Claude daily (work and hobby respectively). And there are days where one or the other just seems to have gotten up on the wrong side of the bed. Or is just being lazy. Or is suddenly super-powered do everything including what i asked it not to. (To be fair, the same thing happens with myself. :/) I am convinced that if I was bench-marking, I would be convinced these are different models on different days. [This conviction may say more about me then about the model.]
- serial_dev 6mo agoThat's also fair, Anthropic lobotomized their services a couple of times already. One week, you are in awe that the tools figure out everything, explain everything, consider everything, produce a clean fix... next week, they are completely useless.
- taegee 6mo agoWhile I partially agree with you, there IS work being done to make the metrics comparable. Eg: https://ghzhang233.github.io/blog/2026/03/05/train-before-test/ https://ghzhang233.github.io/blog/2026/03/05/train-before-te... It just hasn't been widely adopted yet. And it might be in each of their particular interests that it continues to stay so for a while. It's basically like p-hacking.
- cyanydeez 6mo agoUnfortunately, you're probably right, but the cock measuring contest is going to keep escalating because the billionaires and VC backers need to _win_. And the Psychosis is going to produce some horrible collateral damage.
- surgical_fire 6mo agoThis is a problem for OpenAI and Anthropic when they are bleeding money and in desperate need to jack up prices by moving people to their very expensive API. It's very difficult to justify spending on the their models in a world where DeepSeek costs a fraction and Chinese open models exists and they perform as well as what is considered the state of the art, and it only depends on you adjusting how you use them. A couple of days ago I canceled ChatGPT and started to try out DeepSeek. Let's see how it goes.
- Danox 6mo agoCheaper and only 3 to 6 months behind at most.
- mark_l_watson 6mo agoI agree. I have rather constrained use cases for LLMs and the agentic harnesses that I use with them. I try one or two of my use cases with new models or harnesses, make my own often subjective judgements, and largely ignore benchmarks. Blogging and writing in general are a business, or feed other tech adjacent businesses, and a lot of writing about evals is attention getting - nothing wrong with that but there is a lot of noise.