5 ms·
Why I do not believe this shows Anthropic serves folks a worse model: 1. The percentage drop is too low and oscillating, it goes up and down. 2. The baseline
by antirez 8mo ago
Why I do not believe this shows Anthropic serves folks a worse model:
1. The percentage drop is too low and oscillating, it goes up and down.
2. The baseline of Sonnet 4.5 (the obvious choice for when they have GPU busy for the next training) should be established to see Opus at some point goes Sonnet level. This was not done but likely we would see a much sharp decline in certain days / periods. The graph would look like dominated by a "square wave" shape.
3. There are much better explanations for this oscillation: A) They have multiple checkpoints and are A/B testing, CC asks you feedbacks about the session. B) Claude Code itself gets updated, as the exact tools version the agent can use change. In part it is the natural variability due to the token sampling that makes runs not equivalent (sometimes it makes suboptimal decisions compared to T=0) other than not deterministic, but this is the price to pay to have some variability.
- eterm 8mo ago4. The graph starts January 8. Why January 8? Was that an outlier high point? IIRC, Opus 4.5 was released late november.
- littlestymaar 8mo agoOr maybe, juste maybe, that's when they started testing…
- eterm 8mo agoWayback machine has nothing for this site before today, and article is "last updated Jan 29". A benchmark like this ought to start fresh from when it is published. I don't entirely doubt the degradation, but the choice of where they went back to feels a bit cherry-picked to demonstrate the value of the benchmark.
- littlestymaar 8mo agoWhich makes sense, you gotta wait until you get enough data before you can communicate on the said data… If anything it's coherent with the fact that they very likely didn't have data earlier than January the 8th.
- pertymcpert 8mo agoPeople were away for the holidays. What do you want them to do?
- F7F7F7 8mo agoRight after the Holiday double token promotion users felt (perceived) a huge regression in capabilities. I bet that triggered the idea.
- littlestymaar 8mo ago> 1. The percentage drop is too low and oscillating, it goes up and down. How do you define “too low”, they make sure to communicate about the statistical significance of their measurements, what's the point if people can just claim it's “too low” based on personal vibes…
- levkk 8mo agoI believe the science, but I've been using it daily and it's been getting worse, noticeably.
- warkdarrior 8mo agoIs it possible that your expectations are increasing, not that the model is getting worse?
- GoatInGrey 8mo agoPossible, though you eventually run into types of issues that you recall the model just not having before. Like accessing a database or not following the SOP you have it read each time it performs X routine task. There are also patterns that are much less ambiguous like getting caught in loops or failing to execute a script it wrote after ten attempts.
- merlindru 8mo agoyes but i keep wondering if that's just the game of chance doing its thing like these models are nondeterministic right? (besides the fact that rng things like top k selection and temperature exist) say with every prompt there is 2% odds the AI gets it massively wrong. what if i had just lucked out the past couple weeks and now i had a streak of bad luck? and since my expectations are based on its previous (lucky) performance i now judge it even though it isn't different? or is it giving you consistenly worse performance, not able to get it right even after clearing context and trying again, on the exact same problem etc?
- F7F7F7 8mo agoI’ve had Opus struggle on trivial things that Sonnet 3.5 handled with ease. It’s not so much that the implementations are bad because the code is bad (the code is bad). It’s that it gets extremely confused and starts to frantically make worse and worse decisions and questioning itself. Editing multiple files, changing its mind and only fixing one or two. Reseting and overriding multiple batches of commits without so much as a second thought and losing days of work (yes, I’ve learned my lesson). It, the model, can’t even reason with the decisions it’s making from turn to turn. And the more opaque agentic help it’s getting the more I suspect that tasks are being routed to much lesser models (not the ones we’ve chosen via /model or those in our agent definitions) however Anthropic chooses. In these moments I mind as well be using Haiku.
- TIPSIO 8mo agoI too suspect the A/B testing is the prime suspect: context window limits, system prompts, MAYBE some other questionable things that should be disclosed. Either way, if true, given the cost I wish I could opt-out or it were more transparent. Put out variants you can select and see which one people flock to. I and many others would probably test constantly and provide detailed feedback. All speculation though
- F7F7F7 8mo agoWhenever I see new behaviors and suspect I’m being tested on I’ll typically see a feedback form at some point in that session. Well, that and dropping four letter words. I know it’s more random sampling than not. But they are definitely using our codebases (and in some respects our livelihoods) as their guinea pigs.
- samusiam 8mo agoIf that's the case, then as a benchmark operator you'd want to run the benchmark through multiple different accounts on different machines to average over A/B test noise.
- make3 8mo agoIt would be very easy for them to switch the various (compute) cost vs performance knobs down depending on load to maintain a certain latency; you would see oscillations like this, especially if the benchmark is not always run exactly at the same time every day. & it would be easy for them to start with a very costly inference setup for a marketing / reputation boost, and slowly turn the knobs down (smaller model, more quantized model, less thinking time, fewer MoE experts, etc)