7 ms·
There are two possible explanations for this behavior: the model nerf is real, or there's a perceptual/psychological shift. However, benchmarks exist. And I ha
by lukev 11mo ago
There are two possible explanations for this behavior: the model nerf is real, or there's a perceptual/psychological shift.
However, benchmarks exist. And I haven't seen any empirical evidence that the performance of a given model version grows worse over time on benchmarks (in general.)
Therefore, some combination of two things are true:
1. The nerf is psychologial, not actual.
2. The nerf is real but in a way that is perceptual to humans, but not benchmarks.
#1 seems more plausible to me a priori, but if you aren't inclined to believe that, you should be positively intrigued by #2, since it points towards a powerful paradigm shift of how we think about the capabilities of LLMs in general... it would mean there is an "x-factor" that we're entirely unable to capture in any benchmark to date.
- blurbleblurble 11mo agoI'm pretty sure this isn't happening with the API versions as much as with the "pro plan" (loss leader priced) routers. I imagine that there are others like me working on hard problems for long periods with the model setting pegged to high. Why wouldn't the companies throttle us? It could even just be that they just apply simple rate limits and that this degrades the effectiveness of the feedback loop between the person and the model. If I have to wait 20 minutes for GPT-5.1-codex-max medium to look at `git diff` and give a paltry and inaccurate summary (yes this is where things are at for me right now, all this week) it's not going to be productive.
- fakwandi_priv 11mo agoI run the same config but it tends to fly through those commands on the weekends, very noticeable difference. I wouldn’t be surprised that the subscription users have a (much) lower priority. That said I don’t go beyond 70% of my weekly limit so there’s that.
- zsoltkacsandi 11mo ago> The nerf is psychologial, not actual Once I tested this, I gave the same task for a model after the release and a couple weeks later. In the first attempt it produced a well-written code that worked beautifully, I started to worry about the jobs of the software engineers. Second attempt was a nightmare, like a butcher acting as a junior developer performing a surgery on a horse. Is this empirical evidence? And this is not only my experience. Calling this phychological is gaslighting.
- ACCount37 11mo agoNo, it's entirely psychological. Users are not reliable model evaluators. It's a lesson the industry will, I'm afraid, have to learn and relearn over and over again.
- zsoltkacsandi 11mo agoGiving the same prompt resulting in totally different results is not user evaluation. Nor psychological. You cannot tell the customer you are working for as a developer, that hey, first time it did what you asked, second time it ruined everything, but look, here is the benchmark from Antrophic, according to this there is nothing wrong. The only thing that matters and that can evaluate performance is the end result. But hey, the solution is easy: Antrophic can release their own benchmarks, so everyone can test their models any time. Why they don’t do it?
- pertymcpert 11mo agoThe models are non-deterministic. You can't just assume that because it did better before that it was on average better than before. And the variance is quite large.
- zsoltkacsandi 11mo agoNo one talked about determinism. First it was able to do a task, second time not. It’s not that the implementation details changed.
- imiric 11mo agoOr, 2b: the nerf is real, but benchmarks are gamed and models are trained to excel at them, yet fall flat in real world situations.
- metalliqaz 11mo agoI mostly stay out of the LLM space but I thought it was an open secret already that the benchmarks are absolutely gamed.
- davidsainez 11mo agoThere are well documented cases of performance degradation: https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues https://www.anthropic.com/engineering/a-postmortem-of-three-.... The real issue is that there is no reliable system currently in place for the end user (other than being willing to burn the cash and run your own benchmarks regularly) to detect changes in performance. It feels to me like a perfect storm. A combination of high cost of inference, extreme competition, and the statistical nature of LLMs make it very tempting for a provider to tune their infrastructure in order to squeeze more volume from their hardware. I don't mean to imply bad faith actors: things are moving at breakneck speed and people are trying anything that sticks. But the problem persists, people are building on systems that are in constant flux (for better or for worse).
- Wowfunhappy 11mo ago> There are well documented cases of performance degradation: https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues https://www.anthropic.com/engineering/a-postmortem-of-three-... There was one well-documented case of performance degradation which arose from a stupid bug, not some secret cost cutting measure.
- davidsainez 11mo agoI never claimed that it was being done in secrecy. Here is another example: https://groq.com/blog/inside-the-lpu-deconstructing-groq-speed https://groq.com/blog/inside-the-lpu-deconstructing-groq-spe.... I have seen multiple people mention openrouter multiple times here on HN: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=openrouter%20quant&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... Again, I'm not claiming malicious intent. But model performance depends on a number of factors and the end-user just sees benchmarks for a specific configuration. For me to have a high degree of confidence in a provider I would need to see open and continuous benchmarking of the end-user API.
- stingraycharles 11mo ago
- conception 11mo agoThe only time Ive seen benchmark nerfing is I saw one see a drop in performance between 2.5 march preview and release.
- parineum 11mo agoIs there a reason not to think that, when "refining" the models they're using the benchmarks as the measure and it shows no fidelity loss but in some unbenchmarked ways, the performance is worse. "Once a measure becomes a target, it's no longer a useful measure." That's case #2 for you but I think the explanation I've proposed is pretty likely.
- teruakohatu 11mo ago> 1. The nerf is psychologial, not actual. 2. The nerf is real but in a way that is perceptual to humans, but not benchmarks. They could publish weekly benchmarks. To disprove. They almost certainly have internal benchmarking. The shift is certainly real. It might not be model performance but contextual changes or token performance (tasks take longer even if the model stays the same).
- ChadNauseam 11mo agoAnyone can publish weekly benchmarks. If you think anthropic is lying about not nerfing their models you shouldn't trust benchmarks they release anyway.
- teruakohatu 11mo agoI never said they were lying. They haven’t stated that they do not tweak compute, and we know the app is updated regularly.
- yawnxyz 11mo agomoving onto new hardware + caching + optimizations might actually change the output slightly; it'll still pass evals all the same but on the edges it just "feels weird" - and that's what makes it feel like it's nerfed
- data-ottawa 11mo agoThe previous “nerf” was actually several bugs that dramatically decreased performance for weeks. I do suspect continued fine tuning lowers quality — stuff they roll out for safety/jailbreak prevention. Those should in theory buildup over time with their fine tune dataset, but each model will have its own flaws that need tuning out. I do also suspect there’s a bit of mental adjustment that goes in too.
- jaggs 11mo agohttps://www.youtube.com/watch?v=DtePicx_kFY https://www.youtube.com/watch?v=DtePicx_kFY "There's something still not quite right with the current technology. I think the phrase that's becoming popular is 'jagged intelligence'. The fact that you can ask an LLM something and they can solve literally a PhD level problem, and then in the next sentence they can say something so clearly, obviously wrong that it's jarring. And I think this is probably a reflection of something fundamentally wrong with the current architectures as amazing as they are." Llion Jones, co-inventor of transformers architecture
- zbyforgotp 11mo agoThere is something not right with expecting that artificial intelligence will have the same characteristics as human intelligence. (I am answering to the quote)
- jaggs 11mo agoI think he's commenting more on the inconsistency of it, rather than the level of intelligence per se.
- roudaki 11mo agothis. I keep repeating to people to stick to very specific questions with very specific limits and expectations but no... give me 20 pages of phd level text that finds cure for cancer
- csomar 11mo agoThey are nerfed and there is actually a very simple test to prove otherwise: 0 temperature. This is only allowed with the API where you are billed full token prices. Conclusion: It is nerfed unless Claude can prove otherwise.
- Wowfunhappy 11mo agoI don’t understand how you get from the first paragraph to the conclusion.
- camdenreslink 11mo agoAs a personal anecdote, I had a fairly involved application that built up a context with a lot of custom prompting and created a ~1000 word output. I could run my application over and over again to inspect the results. It was fairly reproducible. I was having really nice results with the o4-mini model with high thinking. A little while after GPT-5 came out I revisited my application and tried to continue. The o4-mini results were unusable, while the GPT-5 results were similar to what I had before. I'm not sure what happened to the model in those ~4-5 months I set it down, but there was real degradation.