30 ms·
I'm becoming concerned with the rate at which major software systems seem to be failing as of late. For context, last year I only logged four outages that actua
by aeldidi 11mo ago
I'm becoming concerned with the rate at which major software systems seem to be failing as of late. For context, last year I only logged four outages that actually disrupted my work; this quarter alone I'm already on my fourth, all within the past few weeks. This is, of course, just an anecdote and not evidence of any wider trend (not to mention that I might not have even logged everything last year), but it was enough to nudge me into writing this today (helped by the fact that I suddenly had some downtime). Keep in mind, this isn't necessarily specific to this outage, just something that's been on my mind enough to warrant writing about it.
It feels like resiliency is becoming a bit of a lost art in networked software. I've spent a good chunk of this year chasing down intermittent failures at work, and I really underestimated how much work goes into shrinking the "blast radius", so to speak, of any bug or outage. Even though we mostly run a monolith, we still depend on a bunch of external pieces like daemons, databases, Redis, S3, monitoring, and third-party integrations, and we generally assume that these things are present and working in most places, which wasn't always the case. My response was to better document the failure conditions, and once I did, realize that there was many more than we initially thought. Since then we've done things like: move some things to a VPS instead of cloud services, automate deployment more than we already had, greatly improve the test suite and docs to include these newly considered failure conditions, and generally cut down on moving parts. It was a ton of effort, but the payoff has finally shown up: our records show fewer surprises which means fewer distractions and a much calmer system overall. Without that unglamorous work, things would've only grown more fragile as complexity crept in. And I worry that, more broadly, we're slowly un-learning how to build systems that stay up even when the inevitable bug or failure shows up.
For completeness, here are the outages that prompted this: the AWS us-east-1 outage in October (took down the Lightspeed R series API), the Azure Front Door outage (prevented Playwright from downloading browsers for tests), today’s Cloudflare outage (took down Lightspeed’s website, which some of our clients rely on), and the Github outage affecting basically everyone who uses it as their git host.
- HardwareLust 11mo agoIt's money, of course. No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray.
- lopatin 11mo agoI agree that it's all money. That's why it's always DNS right? > No one wants to pay for resilience/redundancy These companies do take it seriously, on the software side, but when it comes to configurations, what are you going to do: Either play it by ear, or literally double your cloud costs for a true, real prod-parallel to mitigate that risk. It looks like even the most critical and prestigious companies in the world are doing the former.
- macintux 11mo ago> Either play it by ear, or literally double your cloud costs for a true, real prod-parallel to mitigate that risk. There's also the problem that doubling your cloud footprint to reduce the risk of a single point of failure introduces new risks: more configuration to break, new modes of failure when both infrastructures are accidentally live and processing traffic, etc. Back when companies typically ran their own datacenters (or otherwise heavily relied on physical devices), I was very skeptical about redundant switches, fearing the redundant hardware would cause more problems than it solved.
- paulddraper 11mo agoComplexity breeds bugs. Which is why the “art” of engineering is reducing complexity while retaining functionality.
- ForHackernews 11mo agoWhy should they? Honestly most of what we do simply does not matter that much. 99.9% uptime is fine in 99.999% of cases.
- porridgeraisin 11mo agoThis is true. But unfortunately the exact same process is used even for critical stuff (the crowdstrike thing for example). Maybe there needs to be a separate swe process for those things as well, just like there is for aviation. This means not using the same dev tooling, which is a lot of effort.
- suddenlybananas 11mo agoTo be deliberately provocative, LLMs are being more and more widely used.
- blibble 11mo agoimagine what it'll be like in 10 years time Microsoft: the film Idiocracy was not supposed to be a manual
- zdragnar 11mo agoWord on the street is github was already a giant mess before the rise of LLMs, and it has not improved with the move to MS.
- dsagent 11mo agoThey are also in the process of moving most of the infra from on-prem to Azure. I'm sure will see more issues over the next couple months. https://thenewstack.io/github-will-prioritize-migrating-to-azure-over-feature-development/ https://thenewstack.io/github-will-prioritize-migrating-to-a...
- array_key_first 11mo agoI don't know anything about githubs codebase, but as a user, their software has many obvious deficiencies. The most glaring being performance. Oh my God, github performs like absolute shit on large repos and big diffs. Performance issues always scare me. A lot of the time it's indicative of fragile systems. Like with a lot of banking software - the performance is often bad because the software relies on 10 APIs to perform simple tasks. I doubt this is the case with GitHub, but it still makes you wonder about their code and processes. Especially when it's been a problem for many years, with virtually no improvement.
- Tadpole9181 11mo agoTo be deliberately provocative, so is offshoring work.
- roxolotl 11mo agoTo agree with the comments it seems likely it's money which has begun to result in a slow "un-learning how to build systems that stay up even when the inevitable bug or failure shows up."