4 ms·

Anthropic claims that they don't degrade models under load, and the performance issues were a result of a system error: https://status.anthropic.com/incidents/

Anthropic claims that they don't degrade models under load, and the performance issues were a result of a system error: https://status.anthropic.com/incidents/72f99lh1cj2c https://status.anthropic.com/incidents/72f99lh1cj2c That being said, they still have capacity issues on any day of the week that ends in Y. No clue how long would that take to resolve.

pmx 1y ago
Frankly, I don't believe their claims that they don't degrade the models. I know we see models as less intelligent as we get used to them and their novelty wears off but I've had to entirely give up on Claude as a coding assistant because it seems to be incapable of following instructions anymore.
- SparkyMcUnicorn 1y ago
  I'd believe a lot of other claims before believing model degradation was happening. - They admittedly go off of "vibes" for system prompt updates[0] - I've seen my coworkers making a lot of bad config and CLAUDE.md updates, MCP server span, etc. and claiming the model got worse. After running it with a clean slate, they redacted their claims. [0] https://youtu.be/iF9iV4xponk?t=459 https://youtu.be/iF9iV4xponk?t=459
mh- 1y ago
Not nitpicking, but they said: > we never intentionally degrade model quality as a result of demand or other factors Fully giving them the benefit of the doubt, I still think that still allows for a scenario like "we may [switch to quantized models|tune parameters], but our internal testing showed that these interventions didn't materially affect end user experience". I hate to parse their words in this way, because I don't know how they could have phrased it that closed the door on this concern, but all the anecdata (personal and otherwise) suggests something is happening.
- SparkyMcUnicorn 1y ago
  "or other factors" is pretty catch-all in my opinion. > I don't know how they could have phrased it that closed the door on this concern Agreed. A full legal document would probably be the only way to convince everyone.
- ACCount37 1y ago
  "Anecdata" is notoriously unreliable when it comes to estimating AI performance over time. Sure, people complain about Anthropic's AI models getting worse over time. As well as OpenAI's models getting worse over time. But guess what? If you serve them open weights models, they also complain about models getting worse over time. Same exact checkpoint, same exact settings, same exact hardware. Relative LMArena metrics, however, are fairly consistent across time. The takeaway is that users are not reliable LLM evaluators. My hypothesis is that users have a "learning curve", and get better at spotting LLM mistakes over time - both overall and for a specific model checkpoint. Resulting in increasingly critical evaluations over time.
  - yazanobeidi 1y ago
    You’re not wrong, but, I can literally see it get worse throughout the day sometimes, especially recently. Coinciding with Pacific Time Zone business hours. Quantization could be done, not to deliberately make the model worse, but to increase reliability! Like Apple throttling devices - they were just trying to save your battery! After all there are regular outages, and some pretty major ones a handful of weeks back taking eg Opus offline for an entire afternoon.
  - rapind 1y ago
    And yet, people's complaints about Claude Code over the past month and a bit are now justified by Anthropic stating that those complaints caused them to investigate and fix a bunch of issues (while investigating potential more issues with opus). > But guess what? If you serve them open weights models, they also complain about models getting worse over time. Isn't this also anecdotal, or is there data informing this statement? I think you could be partially right, but I also don't think dismissing criticism as just being a change in perspective is correct either. At least some complaints are from power users who can usually tell when something is getting objectively worse (as was the case for some of us Claude Code users recently). I'm not saying we can't fool ourselves too, but I don't think that's the most likely assumption to make.
siva7 1y ago
Then check the news again. They already admitted that due to bugs model output was degraded for over a month
- ACCount37 1y ago
  My link IS that news.
fragmede 1y ago
> Last week, we opened an incident to investigate degraded quality in some Claude model responses. We found two separate issues that we’ve now resolved.

pmx 1y ago

Frankly, I don't believe their claims that they don't degrade the models. I know we see models as less intelligent as we get used to them and their novelty wears off but I've had to entirely give up on Claude as a coding assistant because it seems to be incapable of following instructions anymore.

SparkyMcUnicorn 1y ago
I'd believe a lot of other claims before believing model degradation was happening. - They admittedly go off of "vibes" for system prompt updates[0] - I've seen my coworkers making a lot of bad config and CLAUDE.md updates, MCP server span, etc. and claiming the model got worse. After running it with a clean slate, they redacted their claims. [0] https://youtu.be/iF9iV4xponk?t=459 https://youtu.be/iF9iV4xponk?t=459

mh- 1y ago

Not nitpicking, but they said: > we never intentionally degrade model quality as a result of demand or other factors Fully giving them the benefit of the doubt, I still think that still allows for a scenario like "we may [switch to quantized models|tune parameters], but our internal testing showed that these interventions didn't materially affect end user experience". I hate to parse their words in this way, because I don't know how they could have phrased it that closed the door on this concern, but all the anecdata (personal and otherwise) suggests something is happening.

SparkyMcUnicorn 1y ago
"or other factors" is pretty catch-all in my opinion. > I don't know how they could have phrased it that closed the door on this concern Agreed. A full legal document would probably be the only way to convince everyone.
ACCount37 1y ago
"Anecdata" is notoriously unreliable when it comes to estimating AI performance over time. Sure, people complain about Anthropic's AI models getting worse over time. As well as OpenAI's models getting worse over time. But guess what? If you serve them open weights models, they also complain about models getting worse over time. Same exact checkpoint, same exact settings, same exact hardware. Relative LMArena metrics, however, are fairly consistent across time. The takeaway is that users are not reliable LLM evaluators. My hypothesis is that users have a "learning curve", and get better at spotting LLM mistakes over time - both overall and for a specific model checkpoint. Resulting in increasingly critical evaluations over time.
- yazanobeidi 1y ago
  You’re not wrong, but, I can literally see it get worse throughout the day sometimes, especially recently. Coinciding with Pacific Time Zone business hours. Quantization could be done, not to deliberately make the model worse, but to increase reliability! Like Apple throttling devices - they were just trying to save your battery! After all there are regular outages, and some pretty major ones a handful of weeks back taking eg Opus offline for an entire afternoon.
- rapind 1y ago
  And yet, people's complaints about Claude Code over the past month and a bit are now justified by Anthropic stating that those complaints caused them to investigate and fix a bunch of issues (while investigating potential more issues with opus). > But guess what? If you serve them open weights models, they also complain about models getting worse over time. Isn't this also anecdotal, or is there data informing this statement? I think you could be partially right, but I also don't think dismissing criticism as just being a change in perspective is correct either. At least some complaints are from power users who can usually tell when something is getting objectively worse (as was the case for some of us Claude Code users recently). I'm not saying we can't fool ourselves too, but I don't think that's the most likely assumption to make.

siva7 1y ago

Then check the news again. They already admitted that due to bugs model output was degraded for over a month

ACCount37 1y ago
My link IS that news.

fragmede 1y ago

> Last week, we opened an incident to investigate degraded quality in some Claude model responses. We found two separate issues that we’ve now resolved.