5 ms·
Anyone can explain why do they fail so often and so seriously? Is this inherent to managing LLMs or is their Ops really just that bad?
by cromka 2mo ago
Anyone can explain why do they fail so often and so seriously? Is this inherent to managing LLMs or is their Ops really just that bad?
- vinni2 2mo agoBecause they can’t keep up with the demand?
- tvbusy 2mo agoStruggling with demands is a performance problem, not a reliability problem.
- swe_dima 2mo agooverly high demand can often put a system in an unstable state
- therein 2mo agoOnly if you're incapable of throttling to assure quality of service. Are you saying they have so much demand that even their load balancers were overloaded?
- slopinthebag 2mo agoThere is no point having this discussion. Some people have had their brains genuinely broken by Anthropic and will defend them to the last breath, despite having no clue what they are talking about.
- lelanthran 2mo agoThrottling helps to prevent a 30% failure rate (with a quick recovery time) turning into a 100% failure rate (with practically infinite recovery time), but some users are still going to see failures, and the news is still going to be "Anthropic is down again" because some users are seeing an outage. I'm not really sure how they can avoid getting in the news with "Anthropic is down", TBH. Although, at least by throttling they might get "Anthropic is down for some requests" and not "Anthropic is down for all requests".
- ACCount37 2mo agoIt is a reliability problem - because if the API ends up rejecting 30% of the incoming queries, no one cares if it's 500 Internal Server Error or 529 Overloaded. Having your infrastructure at the knife's edge of load to capacity also means you have no redundancy when something fails.
- cherioo 2mo agoIf this incident had happened during the day I could buy that argument. But it is happening in the dead of the night. Are there really that much people scheduling claude runs over-night, taking more capacity than day-time? (assume most claude users are west coast programmers) My other guess is they are scheduling training runs and their capacity isolation is bad.
- deleted 2mo ago[deleted]
- Recursing 2mo agoI think it's because customers don't seem to mind enough to switch. I really want to support Anthropic because of their commitments to effective altruism, but it's getting harder and harder to justify on technical merits. Sol is just faster, cheaper, and more reliable, and as good as Opus for my use cases
- therein 2mo ago> commitments to effective altruism That's a downside if anything.
- eru 2mo agoDifferent folks have different preferences.
- KingMob 2mo agoYes, some people think effective altruism means helping people here and now as efficiently as possible. Others think the sheer number of potential sentients in the future justify any amount of suffering tolerated in the here and now. Notably, people in the latter group never assume THEY will be the ones suffering for the future.
- eru 2mo ago> Notably, people in the latter group never assume THEY will be the ones suffering for the future. What makes you think so? What evidence do you have? > Yes, some people think effective altruism means helping people here and now as efficiently as possible. Most effective altruists seem pretty relaxed about the 'here' part, but you are right that the 'now' part receives different answers from different subgroups. > Others think the sheer number of potential sentients in the future justify any amount of suffering tolerated in the here and now. There's also the people who care a lot about shrimp. They are less crazy than it seems at first.
- Smaug123 2mo ago
- slopinthebag 2mo agoEverything is vibe coded, would you expect anything different?
- alansaber 2mo agoQuick, release a new claude product then depricate it after 90 days
- ifwinterco 2mo agoBecause coding is solved and there’s no need for software engineers any more. Being charitable to Anthropic though: coding is sort of solved for a very limited definition of “solved”. Software engineering of huge distributed systems at massive scale is very much not solved, and it seems Anthropic don’t have the human capital to do it particularly well
- nixon_why69 2mo agoIf we want to be even more charitable to Anthropic, they're going from nothing to hyperscaler in like 3 years. I'm as happy as anyone to dunk on the general bubble situation and people claiming their technology is too dangerous to share, then something like this happens, it's hilarious. But it's still actually hard to build a service with that much usage and every org has had some outages on their way to figure it out.
- ifwinterco 2mo agoYes I agree, I don't doubt they are dealing with some serious technical challenges. It's not just the enormous rate of growth, LLMs and workflows are also evolving rapidly. Whatever they planned for 1-2 years ago in terms of how they expected their stack to be used is probably already well out of date, so they'll be rearchitecting bits of a production system while it's being used
- chrisjj 2mo agoAnthropic's product is inherently unreliable even when operating at its best. Why would they bother with reliable delivery when their customers are the type clearly not interested in reliability?
- vasco 2mo agoClearly they just want to be down all the time. They have the ability to throttle down traffic. I have no clue why they decide to throttle it down just enough to have the worst of both worlds, throttling and incidents. Throttle more and have zero incidents, what an incredible revelation.
- Barbing 2mo agoThey didn’t want to be down so bad they rented capacity from Elon and uptime improved dramatically. Super flaky before that, they were even playing games with response quality to lighten their load nontransparently. Not sure what the issue is today. But on average, past 90 days, good availability and quality right?