4 ms·
> I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on th
by dahart 5y ago
> I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it."
On one hand a single incident is unlikely to be the end of a large company because companies exist to make money, and also because by the time a company gets to a thousand engineers, it’s engineering function is already centered on managing complexity, and they will have had hundreds of service outages. This is in stark contrast to the single-user free service we’re commenting on, where the owner was already treading water and perhaps just didn’t know it, and where the tradeoffs never had financial upside.
On the other hand, you’re not just right about your prediction, but it has already happened in the broader sense, just maybe not over a single incident. Complexity has already killed many software companies, when the software gets too big to manage and can’t be easily refactored, runs on old languages and tools that stop getting updates, relies on too many services that slowly go out of both fashion and support contracts. The list of companies that got too invested in a single codebase and stuck themselves with the inability to adapt quickly enough due to the weight and complexity of what they built is pretty long and growing.
- nonrandomstring 5y ago> "On one hand a single incident is unlikely to be the end of a large company because companies exist to make money, and also because by the time a company gets to a thousand engineers, it’s engineering function is already centered on managing complexity." Absolutely the companies themselves can be very robust. Some friends and colleagues who work for <big defence> tell me they don't fear for their jobs if a project is crashing. Internally such firms are structured to run dozens of projects, all well compartmentalised, such that if one fails they'll just get moved to another. The company itself is survivable via diversity (plus there's .gov money close at hand to bail out in an emergency). What concerns me about civil resilience is where we've got unexamined mono-cultures like Google Docs or GitHub that become de-facto pillars for countless other businesses. When Trump ordered Adobe to cut-off Venezuela that should have been a resilience wake-up call for everybody (and it prefigures the present Russian situation of course). There could be many reasons (more likely an unfortunate confluence of unrelated upstream reasons) that topple a vital part of public infrastructure. That's why big-tech monopoly, which erodes hybrid resilience, bothers the hell out of me.