6 ms·
Is it me or has there been a very noticeable uptick in large scale infra-level outages lately? AWS, Cloudflare, etc have all been way under whatever SLA they pu
by rco8786 11mo ago
Is it me or has there been a very noticeable uptick in large scale infra-level outages lately? AWS, Cloudflare, etc have all been way under whatever SLA they publish.
- alt227 11mo agoFor me the only silver lining to all these cloud outages is now we know that their published SLA times mean absolutely nothing. The number of 9's used to at least give an indication of intent of reliability, now they are twisted to whatever metric the company wants to represent and dont actually represent guaranteed uptime anywhere.
- bojangleslover 11mo agoSo true. AWS for example gives only platform credits in the event of an outage. Basically no recourse or insurance.
- op00to 11mo agoDoesn’t everyone do that? I’ve never worked for a place that the base policy wasn’t credits. You might have special contract language stating otherwise, but for almost everyone, it’s credits.
- bojangleslover 11mo agoYeah everyone does that. Perhaps it was wrong to single out AWS! None of them should. In any other business, you get your money back.
- op00to 11mo agoI think it's crazy that the SLA just gives you a credit. Even worse! In the contracts I've seen, you have to prove the duration of the outage and sometimes even that there was commercial impact using your own monitoring. What a pain in the ass. It would be easy for me to tell my customers exactly what they need to say to get the maximum, but I've been told not to do that, so I guess it's on them to figure it out.
- dimastopel 11mo agoGCP was down recently as well
- guyfromfargo 11mo agoDon’t forget that Azure was down two weeks ago as well.
- driverdan 11mo agoIt's you. Everything does down once in a while.
- codethief 11mo agoDon't forget Azure Front Door / half of Azure.
- weberer 11mo agoYeah, but that's just standard for Azure.
- AsmaraHolding 11mo agoSome of the other commenters here have posited a "vibe code theory". As the amount of vibe code in production increases, so does the number of bugs and, therefore, the number of outages.
- ACCount37 11mo agoNone of the recent major outages were traced down to "vibe coding" or anything of the sort. They appear to be the kind of misconfigurations and networking fuckups that existed since Internet became more complex than 3 routers.
- Seb-C 11mo agoThe "vibe thinking" trend where people stop using their brain and rely on whatever random output the LLM tells them is harder to diagnose, but it's certainly there and at least as bad as vibe coding.
- Aurornis 11mo agoWhat about the “vibe thinking” trend where people project their own narratives on to every situation, even if the information available shows that it’s a rise in large scale DDoS attacks?
- ACCount37 11mo agoUnfortunately, not a trend. Just human nature. I hope they'll find a fix for that one day.
- davey48016 11mo agoWasn't the recent AWS a race condition that's existed since before vibe coding was a thing?
- ceejayoz 11mo agoHow likely are we to know when a "misconfiguration or networking fuckup" is due to someone asking ChatGPT how to do the task?
- chid 11mo agoit definitely feels like it.
- tristanperry 11mo agoMy theory is a state-sponsored actor targeting some of these services, but maybe that's just too 'tinfoil hat' of me, who knows.
- wepple 11mo agoThere are usually very comprehensive post mortems for these events, and none have suggested that at all
- bflesch 11mo agoThis only amplifies the often-repeated propaganda about the "very powerful" enemies of democracy, who in fact are very fragile dictatorships. There's enough incompetence at tech companies to f up their own stuff.
- rozap 11mo agoMy theory is DNS.
- roxolotl 11mo agoThe theory I’ve heard is holiday deploy freezes coupled with Q4 goals creates pressure to get things in quickly and early. It’s all been in the last month or so which does line up.
- trollbridge 11mo agoCoincidentally, large tech companies have been conducting mass layoffs and claim they're going to rely on AI much more to replace junior developers.
- MaxHoppersGhost 11mo agoAnd they are offshoring roles to lower quality devs.
- gtsop 11mo ago[dead]
- Aurornis 11mo agoThat does seem to be a coincidence, as the recent outages making headlines (including this one according to early reports) have been associated with huge traffic spikes. It seems DDoS are reaching a new level.
- xnx 11mo agoBy similar thinking, you could blame large tech companies if they hired too many juniors.
- teeray 11mo agoJuniors, at least, have the capacity to learn.
- 6c696e7578 11mo ago
- elric 11mo agoI suspect the number of outages is the same, but the number of sites putting all of their eggs into these two baskets has grown considerably.
- mikece 11mo agoAny chance our friend Vladamir is behind this?
- kqr 11mo agoIf it's any guidance, US cyber risk insurance (which covers among other things disruptions due to supplier outages) has continuously dropped in price since Q1 2023, with a handful of percent per year. If you excuse the sloppy plot manually transcribed from market index data: https://i.xkqr.org/cyberinsurancecost.png https://i.xkqr.org/cyberinsurancecost.png
- zulban 11mo agoUnless you're making that determination statistically, it's probably pereidolia. See here: https://behavioralscientist.org/yates-expect-unexpected-why-randomness-doesnt-feel-random-sense-patterns/ https://behavioralscientist.org/yates-expect-unexpected-why-...
- whalesalad 11mo agoSomewhere, at a floating desk behind a wall of lava lamps, in a nyancatified ghostty terminal with 32 different shader plugins installed: You're absolutely right! I shouldn't have force pushed that change to master. Let me try and roll it back. * Confrobulating* Oh no! Cloudflare appears to be down and I cannot revert the change. Why don't you go make a cup of coffee until that comes back. This code is production ready, it's probably just a blip.
- 1zael 11mo agoWell AWS runs on Cloudflare...so thanks Cloudflare team!