8 ms·

A postmortem of three recent issues

381 points by moatmoat 1y ago

moatmoat 1y ago
TL;DR — Anthropic Postmortem of Three Recent Issues In Aug–Sep 2025, Claude users saw degraded output quality due to infrastructure bugs, not intentional changes. The Three Issues 1. *Context window routing error* - Short-context requests sometimes routed to long-context servers. - Started small, worsened after load-balancing changes. 2. *Output corruption* - TPU misconfigurations led to weird outputs (wrong language, syntax errors). - Runtime optimizations wrongly boosted improbable tokens. 3. *Approximate top-k miscompilation* - A compiler bug in TPU/XLA stack corrupted token probability selection. - Occasionally dropped the true top token. Why It Was Hard to Detect - Bugs were subtle, intermittent, and platform-dependent. - Benchmarks missed these degradations. - Privacy/safety rules limited access to real user data for debugging. Fixes and Next Steps - More sensitive, continuous evals on production. - Better tools to debug user feedback safely. - Stronger validation of routing, output correctness, and token-selection.
- sebastiennight 1y ago
  > Privacy/safety rules limited access to real user data for debugging. Do their ToS really limit access to user data (prompt/response)? I don't remember seeing anything to that effect in their terms.
  - mcintyre1994 1y ago
    I’d imagine they have a lot of internal controls, even if ultimately someone at the company can read the data within their terms. It makes sense that the teams debugging stuff wouldn’t have this access immediately.
  - favorited 1y ago
    I know that when you submit a thumbs up/down rating for a response, you need to opt-in to the whole chat conversation being shared with Anthropic.
stellalo 1y ago
Title should be fixed: it’s about Claude models in general, not Claude Code
- dang 1y ago
  Fixed now, thanks! (Submitted title was "Claude Code Degradation: A postmortem of three recent issues".)
bravetraveler 1y ago
> We don't typically share this level of technical detail about our infrastructure, but the scope and complexity of these issues justified a more comprehensive explanation. Layered in aggrandizing. You host a service, people give you money.
- levocardia 1y ago
  No, what that statement means is "we know that if we just say 'we weren't downgrading performance to save money', you won't believe us, so here is a deep dive on the actual reason it happened"
  - bravetraveler 1y ago
    They can still do the deep dive, that is absolutely convincing. They likely did: distracted before I could finish [work, unfortunately - incident of our own] My criticism is it's 'puffy'. The 'scope and complexity' for a public postmortem is 'customer-facing'. Otherwise it's a tree/forest scenario. One might say 'the lady doth protest too much'; this should be routine. It is, elsewhere: see Cloud, Web Hosting, PBX. Pick your decade.
  - pluto_modadic 1y ago
    they're big, and we expect proper behavior out of them when they mess up. that includes public details.
  - FossQuestion 1y ago
    What deep dive? They explained one of the three issues. and never offered a real solution to the problems. their action items list is just "We will test better next time..."
deepdarkforest 1y ago
Wow. Sneaky. They do not even state the rate of impact for the XLA bug afaik, which affected everyone, not just claude code users, very vague. Interesting. Claude code made almost half a billion so far[1] (>500m in ARR and its like 9 months old) , and 30% of all users have been impacted at least once, just from the first routing bug. Scary stuff. Their post mortem is basically "evaluations are hard, we relied on vibe checking, now we are going to have even more frequent vibe checking". I believe it was indeed unintentional, but in the future where investor's money wont come down from the skies, serving distilled models will be very tempting. And you can not be liable to any SLA currently, it's just vibes. I wonder how enterprise vendors are going to deal with this going forward, you cannot just degrade quality without client or vendor even being able to really prove it. [1][https://www.anthropic.com/news/anthropic-raises-series-f-at-usd183b-post-money-valuation https://www.anthropic.com/news/anthropic-raises-series-f-at-...]
- extr 1y ago
  Is your contention that paying for a service entitles you to zero bugs, ever?
  - deepdarkforest 1y ago
    Of course not! But usually, you can quantify metrics for quality, like uptime, lost transactions, response time, throughput etc. Then you can have accountability, and remediate. Even for other bugs, you can often reproduce and show clearly the impact. But in this case, other than internal benchmarks, you cannot really prove it. There is no accountability yet
    - _zoltan_ 1y ago
      why would they publish the data you seek? I would not publish it either. the blog explains what issues they had and how they fixed them. this is good enough.
  - flutas 1y ago
    If you paid for a streaming service and the HD option only worked for a random subset of users, and not you, would you complain? It's a material difference in the product, not just "a bug."
extr 1y ago
> Incorrect routing affected less than 0.0004% of requests on Google Cloud's Vertex AI between August 27 and September 16. Matches my experience. I use CC through our enterprise Vertex AI account and never noticed any degradation. In general it seems like these bugs, while serious, were substantially less prevalent than anecdotal online reports would have you believe. We are really talking about a ~1-2 week window here where most issues were concentrated, a relatively small percentage of total requests and total users impacted.
- ispeaknumbers 1y ago
  I'm not sure if you can claim these were "less prevalent than anecdotal online reports". From their article: > Approximately 30% of Claude Code users had at least one message routed to the wrong server type, resulting in degraded responses. > However, some users were affected more severely, as our routing is "sticky". This meant that once a request was served by the incorrect server, subsequent follow-ups were likely to be served by the same incorrect server. 30% of Claude Code users getting a degraded response is a huge bug.
  - extr 1y ago
    I don't know about you but my feed is filled with people claiming that they are surely quantizating the model, Anthropic is purposefully degrading things to save money, etc etc. 70% of users were not impacted. 30% had at least one message degraded. One message is basically nothing. I would have appreciated if they had released the full distribution of impact though.
    - flutas 1y ago
      That 30% is of ALL users, not users who made a request, important to note the weasel wording there. How many users forget they have a sub? How many get a sub through work and don't use it often? I'd bet a large number tbh based on other subscription services.
      - smca 1y ago
OGEnthusiast 1y ago
Seems like Claude is using TPUs a lot more than I thought. For some reason I thought 90%+ of their capacity was from AWS.
Wowfunhappy 1y ago
> On August 25, we deployed a misconfiguration to the Claude API TPU servers that caused an error during token generation. An issue caused by a runtime performance optimization occasionally assigned a high probability to tokens that should rarely be produced given the context, for example producing Thai or Chinese characters in response to English prompts, or producing obvious syntax errors in code. A small subset of users that asked a question in English might have seen "สวัสดี" in the middle of the response, for example. Can anyone explain to a layperson how this sort of thing is even possible for an LLM? For normal code, of course stupid bugs happen all the time. You accidentally introduce an off-by-one error in a conditional, for example, or add an extra `goto fail`. But LLMs aren't written by humans! Models are trained by automated programs over a period of many months across unfathomably massive data centers. How would a human introduce a bug like the one described in TFA?
- ashdksnndck 1y ago
  There are many layers of human-written code in between you and the weights.
- sailingparrot 1y ago
  LLMs are still executed by code written by humans. In this case, the model ultimately give you a probability distribution over each (~200k) tokens in the vocabulary. It's then up to you to decide how you want to sample the next token, you could for example just always sample the most likely, or to make the output more creative, you can sample randomly from the top-k tokens. This top-k sampling, to make it efficient, is written in XLA and compiled to run directly as a kernel, there was a bug in that kernel, which presumably led to tokens outside of the top-k window be select from times to times.
- Centigonal 1y ago
  LLMs produce a probability distribution for what the next token might be. How you pick the actual word that is printed next from that probability distribution is by using a sampling approach[1]. If your sampling approach is "select the next word randomly from among the top 4 possibilities" and you flip a > sign, you could end up with the behavior described in the OP. [1] Here is an example of two common approaches: https://www.reddit.com/r/AIDungeon/comments/1eppgyq/can_someone_explain_what_top_k_and_top_p_are_and/ https://www.reddit.com/r/AIDungeon/comments/1eppgyq/can_some...
flutas 1y ago
And yet no offers of credits to make things right for the users, for what was essentially degraded performance of what you paid for. I know I'll probably get push back on this, but it left a sour taste in my mouth when I paid for a $200 sub that felt like it was less useful than ChatGPT Plus ($20) at times. Or to summarize: [south park "we're sorry" gif]
- blackqueeriroh 1y ago
  I’m pretty certain if you check the ToS that Anthropic doesn’t guarantee a level of response quality, and explicitly even says there is zero guarantee, even for paid plans. So to be fair, you are getting exactly what you paid for - a non-deterministic set of generated responses of varying quality and accuracy.
- topaz0 1y ago
  Hey, they need that $200 to postpone their inevitable bankruptcy
- kashunstva 1y ago
  I would not count on any compensation. I was a Pro subscriber until the third week of May when I was no longer able to login due to some auto ban. No idea what triggered it but no one from support has bothered to respond to my appeals form submissions. A lost cause.
cyanf 1y ago
> On August 29, a routine load balancing change unintentionally increased the number of short-context requests routed to the 1M context servers. At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected. Interesting, this implies that the 1M context servers performs worst at low context. Perhaps this is due to some KV cache compression, eviction or sparse attention scheme being applied on these 1M context servers?
- kruxigt 1y ago
  [dead]
- kiratp 1y ago
  This is due to RoPE scaling. > All the notable open-source frameworks implement static YaRN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the rope_scaling configuration only when processing long contexts is required. It is also recommended to modify the factor as needed. For example, if the typical context length for your application is 524,288 tokens, it would be better to set factor as 2.0. https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking
- FossQuestion 1y ago
  The key issue is that their post-mortem never explained what went wrong on two out of three issues. All I know is that my requests can now travel along three completely different code paths, each on its own stack and tuned differently. Those optimizations can flip overnight, independent of any model-version bump—so whatever worked yesterday may already be broken today. I really don't get the praise that they are getting for this postmortem, it only made me more annoyed.
behnamoh 1y ago
Reminder that Anthropic is the only AI company that has never released any open-source/weight models.
- arduanika 1y ago
  Sure, but don't you feel safer that way?
  - behnamoh 1y ago
    of course, who wants an open-source Sonnet 3... /s
- _zoltan_ 1y ago
  and?
zer00eyz 1y ago
If you are going to run a non deterministic system on three very different hardware platforms doesn't it behoove you to tell your users where their experience is coming from? Calling the platforms A, B and C might help provide us the insight we're missing to spot incongruous behaviors faster than trying to aggregate more generalized feedback.
HoyaSaxa 1y ago
I’m pretty surprised that Anthropic can directly impact the infra for AWS Bedrock as this article suggests. That goes against AWSs commitments. I’m sure the same is true for Google Vertex but I haven’t digged in there from a compliance perspective before. > Our own privacy practices also created challenges in investigating reports. Our internal privacy and security controls limit how and when engineers can access user interactions with Claude, in particular when those interactions are not reported to us as feedback. Ok makes sense and glad to hear > It remains particularly helpful for users to continue to send us their feedback directly. You can use the /bug command in Claude Code Ok makes sense and I’d expect that a human can then see the context in that case although I hope it is still very explicit to the end user (I’m not a Claude Code user so I cannot comment) > or you can use the "thumbs down" button in the Claude apps to do so This is pretty concerning. I can’t imagine the average person equates hitting this button with forfeiting their privacy.
- _da_ 1y ago
  > This is pretty concerning. I can’t imagine the average person equates hitting this button with forfeiting their privacy. When you click "thumbs down" you get the message "Submitting this report will send the entire current conversation to Anthropic for future improvements to our models." before you submit the report, I'd consider that pretty explicit.
  - HoyaSaxa 1y ago
    Great to hear. I'm not a Claude user and the article did not make it seem that way.
- l1n 1y ago
  (Anthropic employee, speaking in a personal capacity) > I’m pretty surprised that Anthropic can directly impact the infra for AWS Bedrock as this article suggests. We don't directly manage AWS Bedrock deployments today, those are managed by AWS. > I can’t imagine the average person equates hitting this button with forfeiting their privacy. We specify > Submitting this report will send the entire current conversation to Anthropic for future improvements to our models. in the thumbs down modal. Is there a straightforward way to improve this copy?
vlovich123 1y ago
The value of figuring out how to make their LLM serving deterministic might help them track this down. There was a recent paper about how the received wisdom that kept assigning it to floating point associativity actually overlooked the real reasons for non-determinism [1]. [1] https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
- mmaunder 1y ago
  It has a big impact on performance to do determinism. Which leaves using another model to essentially IQ test their models with reporting and alerting.
- ants_everywhere 1y ago
  network traffic and machine load aren't deterministic. I think for the near term, getting full determinism (e.g. for auditing) is going to only be feasible for batch jobs that are not cost sensitive. A google search isn't deterministic. Neither is loading upvote count on social media. It's common advice in distributed systems to have a graceful degradation state instead of becoming unavailable. That wouldn't be possible in a system that's completely deterministic.
  - vlovich123 1y ago
    Network traffic and machine load don’t usually impact the output of a pure (in the CS sense of purity) math function (which is what an LLM is) unless you’ve written your system to be sensitive to that. > to have a graceful degradation state instead of becoming unavailable. That wouldn't be possible in a system that's completely deterministic. What does this even mean? I see no incompatibility between determinism and your ability to perform the same function more slowly. Determinism just means that the output of the system is solely dependent on the inputs - feed the same inputs get the same outputs. If by degraded state you’re intentionally choosing to change your inputs, that doesn’t change the determinism of your system. When it is said that LLMs aren’t deterministic, it’s because the output token is dependent on the inputs context and all other contexts processed in the same batch because the kernels are written non-deterministically. If the kernels were written deterministically (so that the output only depended on your input context), then there wouldn’t be a problem and it also wouldn’t change the ability for the system to degrade; it would be deterministic because capturing the input context and random seed would be sufficient. As it stands you’d have to capture the interim states of the other inputs being processed in the same batch and that interim state problem is what makes it non deterministic. As for Google search, it’s not clear to me it’s non-deterministic. When you Google the exact same thing twice you get exactly the same page of results and selected snippets. That suggests there’s more determinism in the system than you’re giving it credit for.
data-ottawa 1y ago
With all due respect to the Anthropic team, I think the Claude status page[1] warrants an internal code red for quality. There were 50 incidents in July, 40 incidents in August, and 21 so far in September. I have worked in places where we started approaching half these numbers and they always resulted in a hard pivot to focusing on uptime and quality. Despite this I'm still a paying customer because Claude is a fantastic product and I get a lot of value from it. After trying the API it became a no brainer to buy a 20x Max membership. The amount of stuff I've gotten done with Claude has been awesome. The last several weeks have strongly made me question my subscription. I appreciate the openness of this post, but as a customer I'm not happy. I don't trust that these issues are all discovered and resolved yet, especially the load balancing ones. At least anecdotally I notice that around 12 ET (9AM pacific) my Claude Code sessions noticeably drop in quality. Again, I hope the team is able to continue finding and fixing these issues. Even running local models on my own machine at home I run into complicated bugs all the time — I won't pretend these are easy problems, they are difficult to find and fix. [1] https://status.anthropic.com/history https://status.anthropic.com/history
- lumost 1y ago
  I've become extremely nervous about these sudden declines in quality. Thankfully I don't have a production product using AI (yet), but in my own development experience - the model becoming dramatically dumber suddenly is very difficult to work around. At this point, I'd be surprised if the different vendors on openrouter weren't abusing their trust by silently dropping context/changing quantization levels/reducing experts - or other mischievous means of delivering the same model at lower compute.
  - martinald 1y ago
    Openrouter is aware this is happening and flags it now on the UI. It's a real problem.
- ruszki 1y ago
  I don’t know whether they are better or worse than others. One for sure, a lot of companies lie on their status pages. I encounter outages frequently which are not reported on their status pages. Nowadays, I’m more surprised when they self report some problems. Personally, I didn’t have serious problems with Claude so far, but it’s possible that I was just lucky. In my perspective, it just seems that they are reporting outages in a more faithful way. But that can be completely coincidental.
unit149 1y ago
[dead]
woah 1y ago
Vibe coding gone wrong?
- nojs 1y ago
  Hey now, CC is only 80% vibe coded! https://www.reddit.com/r/singularity/comments/1khxwjh/claude_code_wrote_80_of_its_own_code_anthropic_dev/ https://www.reddit.com/r/singularity/comments/1khxwjh/claude...
  - bdangubic 1y ago
    80% of Atlassian employees use Jira :)
stephen_cagle 1y ago
I do wonder what a random dip in quality causes in a long running conversation? Does the conversation recover at a later point, or does the introduction of temporary idiocy permanently affect the rest of the conversation? Statistically, probably likely that the dip occurred at a point that wasn't too important? But what happens if the idiot comes out at a critical point? Kind of reminds me of the two alternate ways that time travel works in sci-fi. Does the small change to the past explode like a fission reaction, or does history heal itself? Anywho, if errors do accumulate, I can see being very pissed off even with temporary idiocy from the model, as it means it poisons the context for the entire rest of the conversation.
- unsupp0rted 1y ago
  Depends how good your competitors are at capitalizing on it. Guess what Sam Altman is good at.
dantodor 1y ago
That is a very good start in sharing some level of information with their users, and kudos to the Anthropic team for doing that. However, I don't see any mention of the longstanding issue in CC of API timeout errors. And, at least for me, it's the most frustrating one.
- lukasb 1y ago
  I almost never see these. Maybe issue is your network?
mvdtnz 1y ago
I don't believe for one second that response quality dropped because of an infrastructural change and remained degraded, unnoticed, for weeks. This simply does not pass the sniff test.
- blackqueeriroh 1y ago
  Can you provide any proof of what you’re saying? Any examples that would bear out what you’re asserting? Anything at all? “I refuse to believe what the people who would know the best said, for no real reason except that it doesn’t feel right” isn’t exactly the level of considered response we’re hoping for here on HN. :)
  - mccoyb 1y ago
    Have you used these tools at all? It's incredibly obvious. It was obvious for weeks during August, where several people posted about degradation on r/ClaudeAI ... There's a thousand and one reasons why a company valued in the billions, with the eyes of the world watching, would not be completely honest in their public response.
    - breakingcups 1y ago
      Google Gemini had an ongoing issue for at least 3 weeks with empty responses and the thinking context filled with absolute nonsense. It was even acknowledged on social media by their head of product but no official communication at all. No status page updates, nada. Apparently it was even a known issue for much longer but they only started fixing it after a config change made the problem much more prevalent.
yomismoaqui 1y ago
This reminds me of the story [1] about Facebook intentionally breaking parts of its Android app for some users (including crashing or disabling functionality), to see how far it could degrade before users stopped using Facebook. According to reports, users did not stop coming back even when the app was broken for hours. A similar thing happened to me when playing some initial version of The Binding of Isaac on Linux, when it was made with Flash. Its performance wasn't the best but I couldn't stop playing. So if people still returns maybe Anthropic has something great going on with Claude Code. [1]: https://www.theguardian.com/technology/2016/jan/05/facebook-deliberately-breaking-android-apps?utm_source=chatgpt.com https://www.theguardian.com/technology/2016/jan/05/facebook-...
Omnipresent 1y ago
Which LLM can generate that timeline event graphic from text?
mulmboy 1y ago
Big missing piece - what was the impact of the degraded quality? Was it 1% worse / unnoticeable? Did it become useless? The engineering is interesting but I'd like to see it tied to actual impact
- cpursley 1y ago
  Significant, check any Claude related thread here over the last month or the Claude Code subreddit. Anecdotally, the degradation has been so bad that I had to downgrade to a month old version - which has helped a lot. I think part of the problem is there as well (Claude Code).
am17an 1y ago
They must really be having a bad time if Anthropic of all labs is willing to share their infra details. On the actual precision bug, it is quite unfortunate on FMA side, numerical issues are often deeply bewildering and no AI can solve them (yet.) Also goes to show, if you are in a super crunch situation like this one (competitor literally eating your lunch every day), you need humans to understand what went wrong and even then it can take weeks to rectify.
ath3nd 1y ago
[dead]
mike_hearn 1y ago
The most interesting thing about this is the apparent absence of unit tests. The test for the XLA compiler bug just prints the outputs, it's more like a repro case than a unit test in the sense that it'd be run by a test harness and have coverage tracked. And the action items are simply to lean more aggressively into evals. Although unit testing an entire LLM is not really feasible right now, all these bugs were in small deterministic parts of the system. Load balancing, top-k probability calculations and so on are all engineered parts no different to other software, and should in principle all be unit testable. At most you need an injectable PRNG. Yes, non-deterministic optimization bugs are awful but I've personally found compiler and database bugs in the past using just regular app test suites. With CI you get a lot of runs so rare events can still surface as long as you investigate flakes. One of my current projects runs every unit test in the same process in parallel, which has proven an excellent and cheap strategy for flushing out rare thread safety issues and database deadlocks. A few days ago I commented on a thread about the Java launch that people often feel Java is "enterprisey" compared to Python because Java code is typically written to be heavily unit testable. A lot of abstraction is driven by the desire for dependency injection, for example. I contrasted that to scripting language culture where I've found testing is often either missing or kinda surface level (e.g. mostly just asserting on types). When I've been learning PyTorch a few years ago I noticed the same thing. The tutorials took you from simple to complex stuff without talking much about how you test or best structure the code. This makes sense for ML research where you don't have a clear goal and success boils down to maxing a score in some kind of eval, but it doesn't make sense for production deployment at scale. I wonder if the AI labs could use more people with SRE and HA SWE background to focus on things like this. I'm kinda skeptical that more aggressive rolling evals-in-prod are the best way to avoid bugs like these happening again.
- vintagedave 1y ago
  I've had to write some detailed prompts and examples to have AI generate the kind of unit tests I want in Python. I've seen the assertions on types alone too. I want assertions on values and more. Even more than that, AI tends to mock _everything_. Mocking is useful, but the more real code a unit test invokes, the better, because the risk is not only the code itself but its interactions, the interface. Yet AI in Python will mock so heavily it barely tests even the code itself, with tautological statements. I've prompted with heavy warnings against mocking and pointing directly at examples of thorough tests as examples. FWIW, Python does have excellent tools for injection and can write really nicely structured code.
itsgrimetime 1y ago
Wish they would have included what the actual failure mode was. I’ve been having issues where Claude Code will just hang after running some tool call, was that caused by one of these bugs?
- nadis 1y ago
  I've noticed the same issue increasingly in the past couple of days; I'm guessing it's unrelated (but still a bug) but hope I'm wrong. The number of times I've had to ask variations of "why did you stop? proceed" has really escalated.
jaiyam 1y ago
Do we know why Google’s Gemini didn’t get affected by the XLA bug? Don't they use approx top-k or mixed precision? Also, TIL that Claude is served in prod at bf16
endymion-light 1y ago
I don't know if this is related, but I've noticed a massive issue recently within web app design where claude will just create random streams of text that displays in the dom. Think it's something specifically related to attempting to use svelte, but definitely a massive degredation that I didn't notice prior to this.
wbnns 1y ago
You're absolutely right!
Razengan 1y ago
Anthropic seems to have a "We're special" syndrome: * No Sign in with Apple on the website, so tough luck if you signed up on iOS via that * Can't buy via in-app purchase so have to give them your card * Can't remove your payment method from your account * Hard to get support after buying a subscription * Asking Claude questions about Claude itself such as privacy just gives you links to their website What even is the benefit of paying for Claude over ChatGPT or Grok that are better in overall UX?
- l1n 1y ago
  > so tough luck if you signed up on iOS via that FYI that you can sign in with your private relay email address. It's annoying though.

381 points by moatmoat 1y ago

TL;DR — Anthropic Postmortem of Three Recent Issues In Aug–Sep 2025, Claude users saw degraded output quality due to infrastructure bugs, not intentional changes. The Three Issues 1. *Context window routing error* - Short-context requests sometimes routed to long-context servers. - Started small, worsened after load-balancing changes. 2. *Output corruption* - TPU misconfigurations led to weird outputs (wrong language, syntax errors). - Runtime optimizations wrongly boosted improbable tokens. 3. *Approximate top-k miscompilation* - A compiler bug in TPU/XLA stack corrupted token probability selection. - Occasionally dropped the true top token. Why It Was Hard to Detect - Bugs were subtle, intermittent, and platform-dependent. - Benchmarks missed these degradations. - Privacy/safety rules limited access to real user data for debugging. Fixes and Next Steps - More sensitive, continuous evals on production. - Better tools to debug user feedback safely. - Stronger validation of routing, output correctness, and token-selection.

sebastiennight 1y ago
> Privacy/safety rules limited access to real user data for debugging. Do their ToS really limit access to user data (prompt/response)? I don't remember seeing anything to that effect in their terms.
- mcintyre1994 1y ago
  I’d imagine they have a lot of internal controls, even if ultimately someone at the company can read the data within their terms. It makes sense that the teams debugging stuff wouldn’t have this access immediately.
- favorited 1y ago
  I know that when you submit a thumbs up/down rating for a response, you need to opt-in to the whole chat conversation being shared with Anthropic.

stellalo 1y ago

Title should be fixed: it’s about Claude models in general, not Claude Code

dang 1y ago
Fixed now, thanks! (Submitted title was "Claude Code Degradation: A postmortem of three recent issues".)

bravetraveler 1y ago

> We don't typically share this level of technical detail about our infrastructure, but the scope and complexity of these issues justified a more comprehensive explanation. Layered in aggrandizing. You host a service, people give you money.

levocardia 1y ago
No, what that statement means is "we know that if we just say 'we weren't downgrading performance to save money', you won't believe us, so here is a deep dive on the actual reason it happened"
- bravetraveler 1y ago
  They can still do the deep dive, that is absolutely convincing. They likely did: distracted before I could finish [work, unfortunately - incident of our own] My criticism is it's 'puffy'. The 'scope and complexity' for a public postmortem is 'customer-facing'. Otherwise it's a tree/forest scenario. One might say 'the lady doth protest too much'; this should be routine. It is, elsewhere: see Cloud, Web Hosting, PBX. Pick your decade.
- pluto_modadic 1y ago
  they're big, and we expect proper behavior out of them when they mess up. that includes public details.
- FossQuestion 1y ago
  What deep dive? They explained one of the three issues. and never offered a real solution to the problems. their action items list is just "We will test better next time..."

deepdarkforest 1y ago

Wow. Sneaky. They do not even state the rate of impact for the XLA bug afaik, which affected everyone, not just claude code users, very vague. Interesting. Claude code made almost half a billion so far[1] (>500m in ARR and its like 9 months old) , and 30% of all users have been impacted at least once, just from the first routing bug. Scary stuff. Their post mortem is basically "evaluations are hard, we relied on vibe checking, now we are going to have even more frequent vibe checking". I believe it was indeed unintentional, but in the future where investor's money wont come down from the skies, serving distilled models will be very tempting. And you can not be liable to any SLA currently, it's just vibes. I wonder how enterprise vendors are going to deal with this going forward, you cannot just degrade quality without client or vendor even being able to really prove it. [1][https://www.anthropic.com/news/anthropic-raises-series-f-at-usd183b-post-money-valuation https://www.anthropic.com/news/anthropic-raises-series-f-at-...]

extr 1y ago
Is your contention that paying for a service entitles you to zero bugs, ever?
- deepdarkforest 1y ago
  Of course not! But usually, you can quantify metrics for quality, like uptime, lost transactions, response time, throughput etc. Then you can have accountability, and remediate. Even for other bugs, you can often reproduce and show clearly the impact. But in this case, other than internal benchmarks, you cannot really prove it. There is no accountability yet
  - _zoltan_ 1y ago
    why would they publish the data you seek? I would not publish it either. the blog explains what issues they had and how they fixed them. this is good enough.
- flutas 1y ago
  If you paid for a streaming service and the HD option only worked for a random subset of users, and not you, would you complain? It's a material difference in the product, not just "a bug."

extr 1y ago

> Incorrect routing affected less than 0.0004% of requests on Google Cloud's Vertex AI between August 27 and September 16. Matches my experience. I use CC through our enterprise Vertex AI account and never noticed any degradation. In general it seems like these bugs, while serious, were substantially less prevalent than anecdotal online reports would have you believe. We are really talking about a ~1-2 week window here where most issues were concentrated, a relatively small percentage of total requests and total users impacted.

ispeaknumbers 1y ago
I'm not sure if you can claim these were "less prevalent than anecdotal online reports". From their article: > Approximately 30% of Claude Code users had at least one message routed to the wrong server type, resulting in degraded responses. > However, some users were affected more severely, as our routing is "sticky". This meant that once a request was served by the incorrect server, subsequent follow-ups were likely to be served by the same incorrect server. 30% of Claude Code users getting a degraded response is a huge bug.
- extr 1y ago
  I don't know about you but my feed is filled with people claiming that they are surely quantizating the model, Anthropic is purposefully degrading things to save money, etc etc. 70% of users were not impacted. 30% had at least one message degraded. One message is basically nothing. I would have appreciated if they had released the full distribution of impact though.
  - flutas 1y ago
    That 30% is of ALL users, not users who made a request, important to note the weasel wording there. How many users forget they have a sub? How many get a sub through work and don't use it often? I'd bet a large number tbh based on other subscription services.
    - smca 1y ago

OGEnthusiast 1y ago

Seems like Claude is using TPUs a lot more than I thought. For some reason I thought 90%+ of their capacity was from AWS.

Wowfunhappy 1y ago

> On August 25, we deployed a misconfiguration to the Claude API TPU servers that caused an error during token generation. An issue caused by a runtime performance optimization occasionally assigned a high probability to tokens that should rarely be produced given the context, for example producing Thai or Chinese characters in response to English prompts, or producing obvious syntax errors in code. A small subset of users that asked a question in English might have seen "สวัสดี" in the middle of the response, for example. Can anyone explain to a layperson how this sort of thing is even possible for an LLM? For normal code, of course stupid bugs happen all the time. You accidentally introduce an off-by-one error in a conditional, for example, or add an extra `goto fail`. But LLMs aren't written by humans! Models are trained by automated programs over a period of many months across unfathomably massive data centers. How would a human introduce a bug like the one described in TFA?

ashdksnndck 1y ago
There are many layers of human-written code in between you and the weights.
sailingparrot 1y ago
LLMs are still executed by code written by humans. In this case, the model ultimately give you a probability distribution over each (~200k) tokens in the vocabulary. It's then up to you to decide how you want to sample the next token, you could for example just always sample the most likely, or to make the output more creative, you can sample randomly from the top-k tokens. This top-k sampling, to make it efficient, is written in XLA and compiled to run directly as a kernel, there was a bug in that kernel, which presumably led to tokens outside of the top-k window be select from times to times.
Centigonal 1y ago
LLMs produce a probability distribution for what the next token might be. How you pick the actual word that is printed next from that probability distribution is by using a sampling approach[1]. If your sampling approach is "select the next word randomly from among the top 4 possibilities" and you flip a > sign, you could end up with the behavior described in the OP. [1] Here is an example of two common approaches: https://www.reddit.com/r/AIDungeon/comments/1eppgyq/can_someone_explain_what_top_k_and_top_p_are_and/ https://www.reddit.com/r/AIDungeon/comments/1eppgyq/can_some...

flutas 1y ago

And yet no offers of credits to make things right for the users, for what was essentially degraded performance of what you paid for. I know I'll probably get push back on this, but it left a sour taste in my mouth when I paid for a $200 sub that felt like it was less useful than ChatGPT Plus ($20) at times. Or to summarize: [south park "we're sorry" gif]

blackqueeriroh 1y ago
I’m pretty certain if you check the ToS that Anthropic doesn’t guarantee a level of response quality, and explicitly even says there is zero guarantee, even for paid plans. So to be fair, you are getting exactly what you paid for - a non-deterministic set of generated responses of varying quality and accuracy.
topaz0 1y ago
Hey, they need that $200 to postpone their inevitable bankruptcy
kashunstva 1y ago
I would not count on any compensation. I was a Pro subscriber until the third week of May when I was no longer able to login due to some auto ban. No idea what triggered it but no one from support has bothered to respond to my appeals form submissions. A lost cause.

cyanf 1y ago

> On August 29, a routine load balancing change unintentionally increased the number of short-context requests routed to the 1M context servers. At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected. Interesting, this implies that the 1M context servers performs worst at low context. Perhaps this is due to some KV cache compression, eviction or sparse attention scheme being applied on these 1M context servers?

kruxigt 1y ago
[dead]
kiratp 1y ago
This is due to RoPE scaling. > All the notable open-source frameworks implement static YaRN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the rope_scaling configuration only when processing long contexts is required. It is also recommended to modify the factor as needed. For example, if the typical context length for your application is 524,288 tokens, it would be better to set factor as 2.0. https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking
FossQuestion 1y ago
The key issue is that their post-mortem never explained what went wrong on two out of three issues. All I know is that my requests can now travel along three completely different code paths, each on its own stack and tuned differently. Those optimizations can flip overnight, independent of any model-version bump—so whatever worked yesterday may already be broken today. I really don't get the praise that they are getting for this postmortem, it only made me more annoyed.

behnamoh 1y ago

Reminder that Anthropic is the only AI company that has never released any open-source/weight models.

arduanika 1y ago
Sure, but don't you feel safer that way?
- behnamoh 1y ago
  of course, who wants an open-source Sonnet 3... /s
_zoltan_ 1y ago
and?

zer00eyz 1y ago

If you are going to run a non deterministic system on three very different hardware platforms doesn't it behoove you to tell your users where their experience is coming from? Calling the platforms A, B and C might help provide us the insight we're missing to spot incongruous behaviors faster than trying to aggregate more generalized feedback.

HoyaSaxa 1y ago

I’m pretty surprised that Anthropic can directly impact the infra for AWS Bedrock as this article suggests. That goes against AWSs commitments. I’m sure the same is true for Google Vertex but I haven’t digged in there from a compliance perspective before. > Our own privacy practices also created challenges in investigating reports. Our internal privacy and security controls limit how and when engineers can access user interactions with Claude, in particular when those interactions are not reported to us as feedback. Ok makes sense and glad to hear > It remains particularly helpful for users to continue to send us their feedback directly. You can use the /bug command in Claude Code Ok makes sense and I’d expect that a human can then see the context in that case although I hope it is still very explicit to the end user (I’m not a Claude Code user so I cannot comment) > or you can use the "thumbs down" button in the Claude apps to do so This is pretty concerning. I can’t imagine the average person equates hitting this button with forfeiting their privacy.

_da_ 1y ago
> This is pretty concerning. I can’t imagine the average person equates hitting this button with forfeiting their privacy. When you click "thumbs down" you get the message "Submitting this report will send the entire current conversation to Anthropic for future improvements to our models." before you submit the report, I'd consider that pretty explicit.
- HoyaSaxa 1y ago
  Great to hear. I'm not a Claude user and the article did not make it seem that way.
l1n 1y ago
(Anthropic employee, speaking in a personal capacity) > I’m pretty surprised that Anthropic can directly impact the infra for AWS Bedrock as this article suggests. We don't directly manage AWS Bedrock deployments today, those are managed by AWS. > I can’t imagine the average person equates hitting this button with forfeiting their privacy. We specify > Submitting this report will send the entire current conversation to Anthropic for future improvements to our models. in the thumbs down modal. Is there a straightforward way to improve this copy?

vlovich123 1y ago

The value of figuring out how to make their LLM serving deterministic might help them track this down. There was a recent paper about how the received wisdom that kept assigning it to floating point associativity actually overlooked the real reasons for non-determinism [1]. [1] https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

mmaunder 1y ago
It has a big impact on performance to do determinism. Which leaves using another model to essentially IQ test their models with reporting and alerting.
ants_everywhere 1y ago
network traffic and machine load aren't deterministic. I think for the near term, getting full determinism (e.g. for auditing) is going to only be feasible for batch jobs that are not cost sensitive. A google search isn't deterministic. Neither is loading upvote count on social media. It's common advice in distributed systems to have a graceful degradation state instead of becoming unavailable. That wouldn't be possible in a system that's completely deterministic.
- vlovich123 1y ago
  Network traffic and machine load don’t usually impact the output of a pure (in the CS sense of purity) math function (which is what an LLM is) unless you’ve written your system to be sensitive to that. > to have a graceful degradation state instead of becoming unavailable. That wouldn't be possible in a system that's completely deterministic. What does this even mean? I see no incompatibility between determinism and your ability to perform the same function more slowly. Determinism just means that the output of the system is solely dependent on the inputs - feed the same inputs get the same outputs. If by degraded state you’re intentionally choosing to change your inputs, that doesn’t change the determinism of your system. When it is said that LLMs aren’t deterministic, it’s because the output token is dependent on the inputs context and all other contexts processed in the same batch because the kernels are written non-deterministically. If the kernels were written deterministically (so that the output only depended on your input context), then there wouldn’t be a problem and it also wouldn’t change the ability for the system to degrade; it would be deterministic because capturing the input context and random seed would be sufficient. As it stands you’d have to capture the interim states of the other inputs being processed in the same batch and that interim state problem is what makes it non deterministic. As for Google search, it’s not clear to me it’s non-deterministic. When you Google the exact same thing twice you get exactly the same page of results and selected snippets. That suggests there’s more determinism in the system than you’re giving it credit for.

data-ottawa 1y ago

With all due respect to the Anthropic team, I think the Claude status page[1] warrants an internal code red for quality. There were 50 incidents in July, 40 incidents in August, and 21 so far in September. I have worked in places where we started approaching half these numbers and they always resulted in a hard pivot to focusing on uptime and quality. Despite this I'm still a paying customer because Claude is a fantastic product and I get a lot of value from it. After trying the API it became a no brainer to buy a 20x Max membership. The amount of stuff I've gotten done with Claude has been awesome. The last several weeks have strongly made me question my subscription. I appreciate the openness of this post, but as a customer I'm not happy. I don't trust that these issues are all discovered and resolved yet, especially the load balancing ones. At least anecdotally I notice that around 12 ET (9AM pacific) my Claude Code sessions noticeably drop in quality. Again, I hope the team is able to continue finding and fixing these issues. Even running local models on my own machine at home I run into complicated bugs all the time — I won't pretend these are easy problems, they are difficult to find and fix. [1] https://status.anthropic.com/history https://status.anthropic.com/history

lumost 1y ago
I've become extremely nervous about these sudden declines in quality. Thankfully I don't have a production product using AI (yet), but in my own development experience - the model becoming dramatically dumber suddenly is very difficult to work around. At this point, I'd be surprised if the different vendors on openrouter weren't abusing their trust by silently dropping context/changing quantization levels/reducing experts - or other mischievous means of delivering the same model at lower compute.
- martinald 1y ago
  Openrouter is aware this is happening and flags it now on the UI. It's a real problem.
ruszki 1y ago
I don’t know whether they are better or worse than others. One for sure, a lot of companies lie on their status pages. I encounter outages frequently which are not reported on their status pages. Nowadays, I’m more surprised when they self report some problems. Personally, I didn’t have serious problems with Claude so far, but it’s possible that I was just lucky. In my perspective, it just seems that they are reporting outages in a more faithful way. But that can be completely coincidental.

unit149 1y ago

[dead]

woah 1y ago

Vibe coding gone wrong?

nojs 1y ago
Hey now, CC is only 80% vibe coded! https://www.reddit.com/r/singularity/comments/1khxwjh/claude_code_wrote_80_of_its_own_code_anthropic_dev/ https://www.reddit.com/r/singularity/comments/1khxwjh/claude...
- bdangubic 1y ago
  80% of Atlassian employees use Jira :)

stephen_cagle 1y ago

I do wonder what a random dip in quality causes in a long running conversation? Does the conversation recover at a later point, or does the introduction of temporary idiocy permanently affect the rest of the conversation? Statistically, probably likely that the dip occurred at a point that wasn't too important? But what happens if the idiot comes out at a critical point? Kind of reminds me of the two alternate ways that time travel works in sci-fi. Does the small change to the past explode like a fission reaction, or does history heal itself? Anywho, if errors do accumulate, I can see being very pissed off even with temporary idiocy from the model, as it means it poisons the context for the entire rest of the conversation.

unsupp0rted 1y ago
Depends how good your competitors are at capitalizing on it. Guess what Sam Altman is good at.

dantodor 1y ago

That is a very good start in sharing some level of information with their users, and kudos to the Anthropic team for doing that. However, I don't see any mention of the longstanding issue in CC of API timeout errors. And, at least for me, it's the most frustrating one.

lukasb 1y ago
I almost never see these. Maybe issue is your network?

mvdtnz 1y ago

I don't believe for one second that response quality dropped because of an infrastructural change and remained degraded, unnoticed, for weeks. This simply does not pass the sniff test.

blackqueeriroh 1y ago
Can you provide any proof of what you’re saying? Any examples that would bear out what you’re asserting? Anything at all? “I refuse to believe what the people who would know the best said, for no real reason except that it doesn’t feel right” isn’t exactly the level of considered response we’re hoping for here on HN. :)
- mccoyb 1y ago
  Have you used these tools at all? It's incredibly obvious. It was obvious for weeks during August, where several people posted about degradation on r/ClaudeAI ... There's a thousand and one reasons why a company valued in the billions, with the eyes of the world watching, would not be completely honest in their public response.
  - breakingcups 1y ago
    Google Gemini had an ongoing issue for at least 3 weeks with empty responses and the thinking context filled with absolute nonsense. It was even acknowledged on social media by their head of product but no official communication at all. No status page updates, nada. Apparently it was even a known issue for much longer but they only started fixing it after a config change made the problem much more prevalent.

yomismoaqui 1y ago

This reminds me of the story [1] about Facebook intentionally breaking parts of its Android app for some users (including crashing or disabling functionality), to see how far it could degrade before users stopped using Facebook. According to reports, users did not stop coming back even when the app was broken for hours. A similar thing happened to me when playing some initial version of The Binding of Isaac on Linux, when it was made with Flash. Its performance wasn't the best but I couldn't stop playing. So if people still returns maybe Anthropic has something great going on with Claude Code. [1]: https://www.theguardian.com/technology/2016/jan/05/facebook-deliberately-breaking-android-apps?utm_source=chatgpt.com https://www.theguardian.com/technology/2016/jan/05/facebook-...

Omnipresent 1y ago

Which LLM can generate that timeline event graphic from text?

mulmboy 1y ago

Big missing piece - what was the impact of the degraded quality? Was it 1% worse / unnoticeable? Did it become useless? The engineering is interesting but I'd like to see it tied to actual impact

cpursley 1y ago
Significant, check any Claude related thread here over the last month or the Claude Code subreddit. Anecdotally, the degradation has been so bad that I had to downgrade to a month old version - which has helped a lot. I think part of the problem is there as well (Claude Code).

am17an 1y ago

They must really be having a bad time if Anthropic of all labs is willing to share their infra details. On the actual precision bug, it is quite unfortunate on FMA side, numerical issues are often deeply bewildering and no AI can solve them (yet.) Also goes to show, if you are in a super crunch situation like this one (competitor literally eating your lunch every day), you need humans to understand what went wrong and even then it can take weeks to rectify.

ath3nd 1y ago

[dead]

mike_hearn 1y ago

The most interesting thing about this is the apparent absence of unit tests. The test for the XLA compiler bug just prints the outputs, it's more like a repro case than a unit test in the sense that it'd be run by a test harness and have coverage tracked. And the action items are simply to lean more aggressively into evals. Although unit testing an entire LLM is not really feasible right now, all these bugs were in small deterministic parts of the system. Load balancing, top-k probability calculations and so on are all engineered parts no different to other software, and should in principle all be unit testable. At most you need an injectable PRNG. Yes, non-deterministic optimization bugs are awful but I've personally found compiler and database bugs in the past using just regular app test suites. With CI you get a lot of runs so rare events can still surface as long as you investigate flakes. One of my current projects runs every unit test in the same process in parallel, which has proven an excellent and cheap strategy for flushing out rare thread safety issues and database deadlocks. A few days ago I commented on a thread about the Java launch that people often feel Java is "enterprisey" compared to Python because Java code is typically written to be heavily unit testable. A lot of abstraction is driven by the desire for dependency injection, for example. I contrasted that to scripting language culture where I've found testing is often either missing or kinda surface level (e.g. mostly just asserting on types). When I've been learning PyTorch a few years ago I noticed the same thing. The tutorials took you from simple to complex stuff without talking much about how you test or best structure the code. This makes sense for ML research where you don't have a clear goal and success boils down to maxing a score in some kind of eval, but it doesn't make sense for production deployment at scale. I wonder if the AI labs could use more people with SRE and HA SWE background to focus on things like this. I'm kinda skeptical that more aggressive rolling evals-in-prod are the best way to avoid bugs like these happening again.

vintagedave 1y ago
I've had to write some detailed prompts and examples to have AI generate the kind of unit tests I want in Python. I've seen the assertions on types alone too. I want assertions on values and more. Even more than that, AI tends to mock _everything_. Mocking is useful, but the more real code a unit test invokes, the better, because the risk is not only the code itself but its interactions, the interface. Yet AI in Python will mock so heavily it barely tests even the code itself, with tautological statements. I've prompted with heavy warnings against mocking and pointing directly at examples of thorough tests as examples. FWIW, Python does have excellent tools for injection and can write really nicely structured code.

itsgrimetime 1y ago

Wish they would have included what the actual failure mode was. I’ve been having issues where Claude Code will just hang after running some tool call, was that caused by one of these bugs?

nadis 1y ago
I've noticed the same issue increasingly in the past couple of days; I'm guessing it's unrelated (but still a bug) but hope I'm wrong. The number of times I've had to ask variations of "why did you stop? proceed" has really escalated.

jaiyam 1y ago

Do we know why Google’s Gemini didn’t get affected by the XLA bug? Don't they use approx top-k or mixed precision? Also, TIL that Claude is served in prod at bf16

endymion-light 1y ago

I don't know if this is related, but I've noticed a massive issue recently within web app design where claude will just create random streams of text that displays in the dom. Think it's something specifically related to attempting to use svelte, but definitely a massive degredation that I didn't notice prior to this.

wbnns 1y ago

You're absolutely right!

Razengan 1y ago

Anthropic seems to have a "We're special" syndrome: * No Sign in with Apple on the website, so tough luck if you signed up on iOS via that * Can't buy via in-app purchase so have to give them your card * Can't remove your payment method from your account * Hard to get support after buying a subscription * Asking Claude questions about Claude itself such as privacy just gives you links to their website What even is the benefit of paying for Claude over ChatGPT or Grok that are better in overall UX?

l1n 1y ago
> so tough luck if you signed up on iOS via that FYI that you can sign in with your private relay email address. It's annoying though.