19 ms·

The Cloudflare outage might be a good thing

• L-four 11mo ago
  It's a tragedy of the commons. Even if you don't use Cloudflare does it matter if no one can pay for your products.
• 0x073 11mo ago
  The outage wasn’t a good thing, since nothing is changing as a result. (How many outages does cloud flare had?)
• charcircuit 11mo ago
  >It's ironic because the internet was actually designed for decentralisation, a system that governments could use to coordinate their response in the event of nuclear war This is not true. The internet was never designed to withstand nuclear war.
  • chasing0entropy 11mo ago
    Arpanet absolutely was designed to be a physically resilient network which could survive the loss of multiple physical switch locations.
  • anonym29 11mo ago
    ARPANET was literally invented during the cold war for the specific and explicit purpose of networked communications resilience for government and military in the event major networking hubs went offline due to one or more successful nuclear attacks against the United States
    • charcircuit 11mo ago
      It literally wasn't. It's an urban myth. >Bob Taylor initiated the ARPANET project in 1966 to enable resource sharing between remote computers. >The ARPANET was not started to create a Command and Control System that would survive a nuclear attack, as many now claim. https://en.wikipedia.org/wiki/ARPANET https://en.wikipedia.org/wiki/ARPANET
      • anonym29 11mo ago
        The stated research goals are not necessarily the same as the strategic funding motivations. The DoD clearly recognized packet-switching's survivability and dynamic routing potential when the US Air Force funded the invention of networked packet switching by Paul Baran six years earlier, in 1960, for which the explicit purpose was "nuclear-survivable military communications". There is zero reason to believe ARPA would've funded the work were it not for internal military recognition of the utility of the underlying technology. To assume that the project lead was told EVERY motivation of the top secret military intelligence committee that was responsible for 100% of the funding of the project takes either a special kind of naïveté or complete ignorance of compartmentalization practices within military R&D and procurement practices. ARPANET would never have been were it not for ARPA funding, and ARPA never would've funded it were it not for the existence of packet-switched networking, which itself was invented and funded, again, six years before Bob Taylor even entered the picture, for the SOLE purpose of "nuclear-survivable military communications". Consider the following sequence of events: 1. US Air Force desires nuclear-survivable military communications, funds Paul Baran's research at RAND 2. Baran proves packet-switching is conceptually viable for nuclear-survivable communications 3. His specific implementation doesn't meet rigorous Air Force deployment standards (their implementation partner, AT&T, refuses - which is entirely expectable for what was then a complex new technology that not a single AT&T engineer understood or had ever interacted with during the course of their education), but the concept is now proven and documented 4. ARPA sees the strategic potential of packet-switched networks for the explicit and sole purpose of nuclear-survivable communications, and decides to fund a more robust development effort 5. They use academic resource-sharing as the development/testing environment (lower stakes, work out the kinks, get future engineers conceptually familiar with the underlying technology paradigms) 6. Researchers, including Bob Taylor, genuinely focus on resource sharing because that's what they're told their actual job is, even though that's not actually the true purpose of their work 7. Once mature, the technology gets deployed for it's originally-intended strategic purposes (MILNET split-off in 1983) Under this timeline, the sole true reason for ARPA's funding of ARPANET is nuclear-survivable military communication, Bob Taylor, being the military's R&D pawn, is never told that (standard compartmentalization practice). Bob Taylor can credibly and honestly state that he was tasked with implementing resource sharing across academic networks, which is true, but was never the actual underlying motivation to fund his research. ...and the myth of "ARPANET wasn't created for nuclear survivability" is born.
  • bblb 11mo ago
    Perhaps. Perhaps not. But it will survive it. It will survive a complete nuclear winter. It's too useful to die, and will be one the first things to be fixed after global annihilation. But Internet is not hosting companies or cloud providers. Internet does not care if they don't build their systems resilient enough and let the SPOFs creep up. Internet does it's thing and the packets keep flowing. Maybe BGP and DNS could use some additional armoring but there are ways around both of them in case of actual emergency.
• chasing0entropy 11mo ago
  Spot on article, but without a call to action. What can we do to combat the migration of society to a centralized corpro-government intertwined entity with no regard for unprofitable privacy or individualism?
  • DANmode 11mo ago
    Learn how to host anything, today.
    • imsurajkadam 11mo ago
      Even if you learn to Host, there are many other services that are going to get relied on those centralised platforms, so if you are thinking to Host, every single thing on your own, then it is going to be more work than you can even imagine and definitely super hard to organise as well
      • DANmode 11mo ago
        Anything.
    • randallsquared 11mo ago
      Have you tried that? I gave up on hosting my own email server seven or eight years ago, after it became clear that there would be an endless fight with various entities to accept my mail. Hosting a webserver without the expectation that you'll need some high powered DDOS defense seems naive, in the current day, and good luck doing that with a server or two.
      • IgorPartola 11mo ago
        I have never hosted my own email. It took me roughly a day to set it up on a vanilla FreeBSD install running on Vultr’s free tier plan and it has been running flawlessly for nearly a year. I did not use AI at all, just the FreeBSD, Postfix, and Dovecot’s handbooks. I do have a fair bit of Linux admin and development experience but all in all this has been a weirdly painless experience. If you don’t love this approach, Mail-in-a-box works incredibly well even if the author of all the Python code behind it insists on using tabs instead of spaces :) And you can always grab a really good deal from a small hosting company, likely with decades of experience in what they do, via LowEndBox/LowEndTalk. The deal would likely blow AWS/DO/Vultr/Google Cloud out of the water in terms of value. I have been snagging deals from there for ages and I lost a virtual host twice. Once was a new company that turned out to be shady and another was when I rented a VPS in Cairo and a revolution broke out. They brought everything back up after a couple of months. For example I just bought a lifetime email hosting system with 250GB of storage, email, video, full office suite, calendar, contacts, and file storage for $75. Configuration here is down to setting the DNS records they give you and adding users. Company behind it has been around for ages and is one of the best regarded in the LET community.
• theideaofcoffee 11mo ago
  > They [outages] can force redundancy and resilience into systems. They won’t until either the monetary pain of outages becomes greater than the inefficiency of holding on to more systems to support that redundancy, or, government steps in with clear regulation forcing their hand. And I’m not sure about the latter. So I’m not holding my breath about anything changing. It will continue to be a circus of doing everything on a shoestring because line must go up every quarter or a shareholder doesn’t keep their wings.
  • morshu9001 11mo ago
    That's ok though, not every website needs 5 9s
• krick 11mo ago
  It would be a good thing, if it would cause anything to change. It obviously won't. As if a single person reading this post wasn't aware that the Internet is centralized, and couldn't name specifically a few sources of centralization (Cloudflare, AWS, Gmail, Github). As if it's the first time this happens. As if after the last time AWS failed (or the one before that, or one before…) anybody stopped using AWS. As if anybody could viably stop using them.
  • captainkrtek 11mo ago
    > It would be a good thing, if it would cause anything to change. It obviously won't. I agree wholeheartedly. The only change is internal to these organizations (eg: CloudFlare, AWS) Improvements will be made to the relevant systems, and some teams internally will also audit for similar behavior, add tests, and fix some bugs. However, nothing external will change. The cycle of pretending like you are going to implement multi-region fades after a week. And each company goes on continuing to leverage all these services to the Nth degree, waiting for the next outage. Not advocating that organizations should/could do much, it's all pros/cons. But the collective blast radius is still impressive.
    • chii 11mo ago
      the root cause is customers refusing to punish these downtime. Checkout how hard customers punish blackouts from the grid - both via wallet, but also via voting/gov't. It's why they are now more reliable. So unless the backbone infrastructure gets the same flak, nothing is going to change. After all, any change is expensive, and the cost of that change needs to be worth it.
      • MikeNotThePope 11mo ago
        Is a little downtime such a bad thing? Trying to avoid some bumps and bruises in your business has diminishing returns.
        • aaron_m04 11mo ago
• timenotwasted 11mo ago
  "Embrace outages, and build redundancy." — It feels like back in the day this was championed pretty hard especially by places like Netflix (Chaos Monkey) but as downtime has become more expected it seems we are sliding backwards. I have a tendency to rely too much on feelings so I'm sure someone could point me to some data that proves otherwise but for now that's my read on things. Personally, I've been going a lot more in on self-hosting lots of things I used to just mindlessly leave on the cloud.
  • bcrl 11mo ago
    I have cell phone calls regularly drop during tower handoffs, and codec errors that result in a blast of static upon answering a call. I can't remember a single time I had a phone call fail on the old PSTN built out of DMS10 and DMS100s locally (well, until we lost all trunks due to a fibre issue a couple of weeks ago on November 10th -- the incumbent didn't notice the outage which started at ~3:20am until ~9:30am, and it wasn't fixed until 17:38). One time when I was a teenager in the '90s, a friend and I had a 14 hour call using landlines. The modern tech stack is disappointing in its lack of reliability. Complexity is the root of all evil.
• stroebs 11mo ago
  The problem is far more nuanced than the internet simply becoming too centralised. I want to host my gas station network’s air machine infrastructure, and I only want people in the US to be able to access it. That simple task is literally impossible with what we have allowed the internet to become. FWIW I love Cloudflare’s products and make use of a large amount of them, but I can’t advocate for using them in my professional job since we actually require distributed infrastructure that won’t fail globally in random ways we can’t control.
  • Fnoord 11mo ago
    Literally impossible? On the contrary; Geofencing is easy. I block all kind of nefarious countries on my firewall, and I don't miss them (no loss not being able to connect to/from a mafia state like Russia). Now, if I were to block FAMAG... or Cloudflare...
    • stroebs 11mo ago
      Yes, literally impossible. The barrier to entry for anyone on the internet to create a proxy or VPN to bypass your geofencing is significantly lower than your cost to prevent them.
      • Aurornis 11mo ago
        I don’t even understand where this line of reasoning is going. Did you want a separate network blocked off from the world? A ban on VPNs? What are we supposed to believe could have been disallowed to make this happen?
        • bigiain 11mo ago
          There are a lot of lists around for known VPN endpoints and datacenter IP address ranges, that people use to reduce error rates in ip address to location lookups. That cannot possibly itself be 100% effective, but it can probably drop the error rate of semi-technical users switching their VPN location to circumvent your geo blocking by an order of magnitude or two. It certainly won't stop a sufficiently motivated technical of malicious user.
• tonyhart7 11mo ago
  I don't like this argument since you can applied this argument to google,microsot,aws,facebook etc Tech world is dominated by US company and what is alternative to most of these service???? its a lot fewer than you might think and even then you must make a compromise in certain areas
• throwaway81523 11mo ago
  Now just wait til every country on earth really does replace most of its employees with ChatGPT... and then OpenAI's data center goes offline with a fiber cut or something. All work everywhere stops. Cloudflare outage is nothing compared to that.
  • DeathArrow 11mo ago
    That's why it's better to have redundancy. Hire Claude and Deepseek, too.
  • teiferer 11mo ago
    > goes offline with a fiber cut If a fiber cut brings your network down then you have fundamental network design issues and need to change hiring practices.
  • delaminator 11mo ago
    That was this outage. ChatGPT and Claude are both behind Clouflare’s bot detection. You couldn’t log into either Web frontends. And the error message said you were blocking them. We had support tickets coming in demanding to know why ChatGPT was being blocked. We also couldn’t log into our supplier’s B2B system to place our customer orders. So all the advice of “just self host” is moot when you’re in a food web.
• oidar 11mo ago
  I wonder what would life without cloudflare look like? What practices would fill the gaps if a company didn't - or wasn't allowed to -- satisfy the the concerns that cloudflare fills.
  • immibis 11mo ago
    Pretty much exactly like it does now but with less captchas. Fun fact: Headless browsers can easily pass cloudflare captchas automatically. They're not actually captchaing - they're just a placebo. You just need to be coming from a residential IP address and using a real browser.
    • hombre_fatal 11mo ago
      > Pretty much exactly like it does now but with less captchas. This just isn't true. e.g. I saw a 30x increase in traffic on my forum due to AI bots that I had to use CF to block. CF is mainly empowered by the naive ideals of the internet's design that never built-in countermeasures against bad actors. You're expected to just deal with it yourself somehow. And that means outsourcing it, especially as residential IP address botnets on unlimited ISP data plans become cheaper and cheaper. Just ask yourself why web hosting providers themselves can't offer services at CF's level. It's because it's too hard of a problem even for them.
      • immibis 11mo ago
        You didn't have to use CF to block them. You chose to use CF to block them. How was your experience with Anubis or https://git.gammaspectra.live/git/go-away https://git.gammaspectra.live/git/go-away? Or you could simply... serve the requests. If your normal traffic is only, like, 1 request per minute, then 30x that is still pretty low and there's no actual reason to worry about it. Web hosting providers don't offer bot blockers because first, they have no reason to care, and second, they can serve the requests, and third, some of them want to upsell you on bandwidth (you should prefer the ones with unmetered bandwidth). BTW AFAIK there's still zero evidence that the massive DDoS wave has anything at all to do with AI. It could be, say, one of Russia's many small avenues of trying to break the West, or Cloudflare trying to get more business, or the NSA trying to make Cloudflare get more business because it's tapped into Cloudflare.
• zie1ony 11mo ago
  My friend wasn't able to do RTG during the outage. They had to use ultrasound machine on his broken arm to see inside.
  • Aurornis 11mo ago
    > My friend wasn't able to do RTG during the outage. What is RTG?
    • soni96pl 11mo ago
      X-ray
    • digestives 11mo ago
      X-ray, in some languages (like Polish) the abbreviation comes from https://en.wikipedia.org/wiki/Roentgen_(unit) https://en.wikipedia.org/wiki/Roentgen_(unit)
    • teiferer 11mo ago
      Wilhelm Röntgen, Nobel Prize in 1901, experimentally discovered X-rays.
    • deleted 11mo ago
      [deleted]
• Surac 11mo ago
  The thing I learned from the incident is that rust offer a unpack function. It puzzles me why the hell they build such a function in the first place.
  • aw1621107 11mo ago
    > It puzzles me why the hell they build such a function in the first place. One reason is similar to why most programming languages don't return an Option<T> when indexing into an array/vector/list/etc. There are always tradeoffs to make, especially when your strangeness budget is going to other things.
• almosthere 11mo ago
  how many people are still on us-east-1
  • mcny 11mo ago
    My old employer used azure. It irritated me to no end when they said we must rename all our resources to match the convention of naming everything US East as "eu-" because (Eastern United States I guess) A total clown show
• 0xbadcafebee 11mo ago
  Centralization has nothing to do with the problems of society and technology. And if you think the internet is all controlled by just a couple companies, you don't actually understand how it works. The internet is wildly decentralized. Even Cloudflare is. It offers tons of services, all of which are completely optional and can be used individually. You can also stop using them at any time, and use any of their competitors (of which there are many). If, on the off chance, people just get "addicted" to Cloudflare, and Cloudflare's now-obviously-terrible engineering causes society to become less reliable, then people will respond to that. Either competitors will pop up, or people will depend on them less, or governments will (finally!) impose some regulations around the operation of technical infrastructure. We have actually too much freedom on the Internet. Companies are free to build internet systems any way they want - including in very unreliable ways - because we impose no regulations or standards requirements on them. Those people are then free to sell products to real people based on this shoddy design, with no penalty for the products falling apart. So far we haven't had any gigantic disasters (Great Chicago Fire, Triangle Shirtwaist Factory Fire, MGM Grand Hotel Fire), but we have had major disruptions. We already dealt with this problem in the rest of society. Buildings have building codes, fire codes, electrical codes. They prescribe and require testing procedures, provide standard building methods to ensure strength in extreme weather, resist a spreading fire long enough to allow people to escape, etc. All measures to ensure the safety and reliability of the things we interact with and depend on. You can build anything you want - say, a preschool? - but you aren't allowed to build it in a shoddy manner. We have that for physical infrastructure; now we need it for virtual infrastructure. A software building code.
  • DeathArrow 11mo ago
    Centralization means having a single point of failure for everything. If your government, mobile phone or car stops working, it doesn't mean all governments, all cars and all mobile phones stop working. Centralization makes mass surveillance easier, makes selectively denying of service easier. Centralization also means that once someone hacks into the system, he gains access to all data, not just a part of it.
• vasco 11mo ago
  I'll die on the hill that centralization is more efficient than decentralization and that rare outages of hugely centralized systems that are otherwise highly reliable are much better than full decentralization with much worse reliability. In other words, when AWS or Cloudflare go down it's catastrophic in the sense that everyone sees the issues at the same time, but smaller providers usually have much more ongoing issues, that just happen to be "chronic" vs "acute" pains.
  • Xelbair 11mo ago
    >I'll die on hill that hyperoptimized systems are more efficient than anti-fragile. Of course they are, the issue is what level of failure were going to accept.
  • GeneralMaximus 11mo ago
    Efficient in terms of what, exactly? There are multiple dimensions to this problem. Putting everything behind Cloudflare might give you better uptime, reliability, performance, etc. but it also has the effect of centralizing power into the hands of a single entity. Instead of twisting the arms of ten different CXOs, your local politician now only needs to twist the arm of a single CXO to knock your entire business off the internet. I live in India, where the government has always been hostile to the ideals of freedom of speech and expression. Complete internet blackouts are common in several states, and major ISPs block websites without due process or an appeals mechanism. Nobody is safe from this, not even Github[1]. In countries like India, decentralization is a preventative measure. [1] https://en.wikipedia.org/wiki/Censorship_of_GitHub#India https://en.wikipedia.org/wiki/Censorship_of_GitHub#India And I'm not even going to talk about abuse of monopoly power and all that. What happens when Cloudflare has their Apple moment? When they jack up their prices 10x, or refuse to serve customers that might use their CDNs to serve "inappropriate" content? When the definition of "inappropriate" is left fuzzy, so that it applies to everything from CSAM to political commentary? No thanks.
    • deleted 11mo ago
      [deleted]
    • 11mo ago
• rzerowan 11mo ago
  So were going backwards to a world where there are basically 5 computers running everything and everyone is basically accessing the world through a dumb terminal.Even though the digital slab in our pockets has more compute than a roomful of the early gen devices. Hopefully critical infrashifts back to managed metal or private clouds - dont see it though with the last decades of cloud evangalism to move all legacy systems to the cloud doesnt look like reversing anytime soon.
  • zwnow 11mo ago
    I agree considering all the Cloudflare AWS Azure apologists I see all around... Learning AWS already is the #1 tip on social media to "become employed as a dev in 2025 guaranteed" and I always just sigh when seeing this. I wouldnt touch it with a stick.
  • fragmede 11mo ago
    Yeah it's crazy to realize it takes a room of electronics for me to get my (g)mail. The more things change, the more they stay the same, eh?
• deleted 11mo ago
  [deleted]
• tomschwiha 11mo ago
  For me personally I didn't notice the downtime in the first hour or so. When using some website assets were not loading, but that's it. Turnstile outage maybe impacted me most. Could be because I'm EU based and Cloudflare is not "so" widespread here as in other parts of the world.
• notepad0x90 11mo ago
  Does the author of this post not see the irony of posting this content on Github? My counter argument is that "centralization" in a technical sense isn't about what company owns things but how services are operated. Cloudflare is very decentralized. Furthermore, I've seen regional outages caused by things like anchors dropped by ships in the wrong place, a shark eating a cable. Regional power outages caused by squirrels,etc... outages happen. If everyone ran their own server from their own home, AT&T or Level3 could have an outage and still take out similar swathes of the internet. With CDNs like cloudflare, if Level3 had an outage, your website won't be down because your home or VPS host's upstream transit happens to be Level3 (or whatever they call themselves these days) because your content (at least static) is cached globally. The only real reasonable alternative is something like ipfs, web3 and similar talk. Cloudflare has always called itself a content transport provider, think of it as such. But also, Cloudflare is just one player, there are several very big players. Every big cloud provider has a competing product, not to mention companies like Akamai. People are rage posting about cloudflare, especially because it has made CDNs accessible to everyone. You can easily setup a free cloudflare account and be on your merry way. This isn't something you should be angry about. You're free to pay for any number of other cdns, many do. If you don't like how Cloudflare has so much market share, then come up with a similarly competitive alternative and profit. Just this HN thread alone is enough for me to think there is a market for more players. Or, just spread the word about the competition that exists today. Use frontdoor, cloudfront, netlify, flycdn, akamai,etc... It's hardly a monopoly.
  • niutech 11mo ago
    Where are already decentralized networks: IPFS (e.g. https://fleek.xyz https://fleek.xyz or https://pinme.eth.limo https://pinme.eth.limo), BlueSky, Web3 protocols with HTTP gateways. Why don't users switch to them?
  • deleted 11mo ago
    [deleted]
  • zero_shift 11mo ago
    As explained in other comments, I only used Gist because publishing to my own site failed. So GH is my redundancy :-)
• jcattle 11mo ago
  "The Cloudflare outage was a good thing [...] they're a warning. They can force redundancy and resilience into systems." - he says. On Github.
  • Afforess 11mo ago
    Thanks for doing the meme! https://knowyourmeme.com/memes/we-should-improve-society-somewhat https://knowyourmeme.com/memes/we-should-improve-society-som... You are very intelligent!
    • jcattle 11mo ago
      That's fair. However I don't think I would have wrote that if those thoughts were shared on a blogging platform. Most blogging platforms do not qualify as critical infrastructure. GitHub with all its CI/CD and supply chain attacks does. There is a certain particular irony of this being written on critical (centralized) infrastructure without any apparent need. Maybe it was intended, maybe not, in any case I found it funny.
      • rkomorn 11mo ago
        I agree. I think the whole point is someone like TFA author has a pretty broad choice of places they can choose to publish this and choosing GitHub is somewhat ironic. Reminds me of the guy who posted an open letter to Mark Zuckerberg like "we are not for sale" on LinkedIn, a place that literally sells access to its users as their main product.
        • deleted 11mo ago
          [deleted]
    • 76684546548070 11mo ago
      Go back to reddit
  • zero_shift 11mo ago
    I published it as a Gist because my own blog deployment pipeline was in a non-functioning state.
• nicman23 11mo ago
  i hate that i cannot just scrape things for me usage and i have to use things like camufox instead of curl
• torginus 11mo ago
  What happens if you don't use Cloudflare and just host everything on a server? Can't you run a website like that if you don't host heavy content? How common are DDOS attacks anyway, and aren't there local (to the server), that analyze user behavior to a decent accuracy (at least it can tell they're using a real browser and behaving more or less like a human would, making attacks expensive). Can't you buy a list of ISP ranges from a GeoIP provider (you can), at least then you'd know which addresses belong to real humans. I don't think botnets are that big of a problem (maybe in some obscure places of the world, but you can temp rangeban a certain IP range, if there's a lot of suspicious traffic coming from there). If lots of legit networks (as in belonging to people who are paying an ISP for their network connections) have botnets, that's means most PCs are compromised, which is a much more severe issue.
  • dijit 11mo ago
    Yeah, you can. Lots of people use raspberry pi’s for this, which is a smidge anaemic for some decent load (HN Hug Of Death)- even an Intel N100 is more grunt, for context. This makes people think that their self hosting setup can never handle HN load; because when they see people talking about self hosting the site goes down.
    • rainonmoon 11mo ago
      Most people shouldn't use a Pi because most people can't configure a web server securely. A VPS would be a better option for just about everybody trying to "self-host" whether they put Cloudflare in front of it or not.
      • dijit 11mo ago
        in both cases you're setting up a webserver. I guess you're concerned about lateral network movement? Justified, but as long as it's patched it's going to be just as secure.
        • rainonmoon 11mo ago
• miki123211 11mo ago
  I don't know how many times I need to say this, but I will die on this hill. Centralized services don't decrease redundancy. They're usually far more redundant than whatever homegrown solution you can come up with. The difference between centralized and homegrown is mostly psychological. We notice the outages of centralized systems more often, as they affect everything at the same time instead of different systems at different times. This is true even if, in a hypothetical world with no centralization, we'd have more total outage time than we do now. If your gas station says "closed" due to a problem that only affects their own networks, people usually go "aah they're probably doing repairs or something", and forget about the problem 5 minutes later. If there's a Cloudflare outage... everybody (rightly) blames the Cloudflare outage. Where this becomes a problem is when correlated failures are actually worse than uncorrelated ones. If Visa goes down, it's better if Mastercard stays up, because many customers have both and can use the other when one doesn't work. In some ways, it's better to have 30 mins of Visa outages today and 30 mins of Mastercard outages tomorrow, than to have just 15 mins of correlated outages in one day.
  • dgan 11mo ago
    > Centralized services don't decrease redundancy Alright, but it creates a failure correlation where previously there was none
    • silvestrov 11mo ago
      Have you ever heard of the "sendmail worm", aka Morris Worm ? https://en.wikipedia.org/wiki/Morris_worm https://en.wikipedia.org/wiki/Morris_worm You can definitely have failure correlation without having centralized services.
  • masfuerte 11mo ago
    In my experience services aren't failing due to a lack of redundancy but due to an excess of complexity. With the move to the cloud we are continually increasing both redundancy and complexity and this is making the problem worse. I have a cheap VPS that has run reliably for a decade except for a planned hour of downtime. Which was in the middle of the night when no-one cared. Amazon is more reliable in theory. My cheap VPS is more reliable in practice.
• chrisjj 11mo ago
  True title: The Cloudflare outage was a good thing
• joeblubaugh 11mo ago
  meta: why are we rewriting such anodyne titles? “was” -> “might be” undermines the author's point
• mrasong 11mo ago
  Yeah, when it went down, a bunch of the sites I use every day just stopped working. That’s when I realized it’s basically one of the backbone pieces of the entire internet.
• ovo101 11mo ago
  Outages like this highlight just how much of the internet’s resilience depends on a single provider. In a way, it’s a healthy reminder: if one company’s hiccup can take down half the web, maybe we’ve over‑centralized. A “good thing” only if it sparks more serious conversations about redundancy, multi‑provider strategies, and reducing monoculture risk. Otherwise, we’ll just keep repeating the same failure modes at larger scales.
• rafaelcosta 11mo ago
  I don't get why this applies on the Cloudflare outage but not on the AWS ones... I'd argue that the big cloud providers are WAY more impactful when they go down than Cloudflare. The only difference is that the average techie uses Cloudflare more and sees the impact more, but this point was already there before...
• SirMaster 11mo ago
  If these systems are as important as they say, it's surprising to me that they are not built with backups and redundancies in place like other mission critical things are engineered and built with.
• tjwebbnorfolk 11mo ago
  Every HN comment seems to say the same thing: downtime is inexcusable and the centralization of these services is ruining the internet. I still don't see the big deal. 12 hours of downtime once every couple years isn't the end of the world. So people can't log into their bank website for a few hours -- banks used to only be open for like 4 hours a day and somehow we all survived. Twitter is down? Oh what a tragedy. Customers get some refunds, Cloudflare fixes the issue, and people move on with life. Cars still break down occasionally after 100+ years of engineering for reliability and safety. The power still goes out every now and then. Cook on the stove. The cost of making everything perfect all the time just isn't worth it. I run my own servers on my own network and do not use Cloudflare. My stuff goes down too. And it's "decentralized" in the way you think the internet "should" be, which entails its own risks. So what do you all want, exactly? A public lashing of every developer at Cloudflare who pushes a bug to prod? A congressional investigation? I just don't understand the outrage here. Stuff breaks occasionally. Get used to it, and design accordingly.
  • joshuamcginnis 11mo ago
    From a consumers perspective, that makes sense. From a business's perspective, downtime can mean significant loss of revenue or new business opportunity.
    • tjwebbnorfolk 11mo ago
      The costs of perfection are much, much greater. Are you willing to pay 2-3x the cost of everything to go from 99.999% to 100.0000000% uptime? Probably the only thing in existence with 100.00% uptime are our nuclear missile command and control systems. Like, even my pen runs out of ink sometimes. It's just crazy how hard it is to have stuff work all the time.
      • blamestross 11mo ago
        I feel obligated to point out that basically no commercial service that relies on a big tech company has better than 99.99% uptime anymore. Your example isn't just hyperbolic, it avoid the actual problem. It isn't that "a bit more reliable" is "nontrivial less reliable than 5 years ago."
• YmiYugy 11mo ago
  It's worth considering the counter factual. Let's say there would be a few dozen semi popular DDoS services. Would that be better? Some assumptions: The services would be slightly less effective and also have worse downtimes. You could argue that Cloudflare is coasting on a monopoly and that competition would drive them to improve, but I'm pretty confident that DDoS protection it one of those things were having a large network to absorb attacks and a large team to monitor them if very valuable. I submit as evidence that Cloudflare has been doing well despite the 3 big cloud providers offering DDoS protection. So what would be the result of a highly decentralized but slightly worse and less reliable DDoS protection? I'd argue that for a lot of things this wouldn't be an improvement. Cloudflare being so dominant means lot's of things go down simultaneously. But that only matters for fungible services, e.g. if a schools education portal goes down, it doesn't matter if all the other education portals are also down. There are cases where it matters like the tyre pumps. I'd argue that these devices have no reason to be reliant on an online connection to begin with. I think cloud services as a whole have massively improved the reliability of internet services. In almost all cases reducing the overall amount of outages is a higher priority than preventing outage correlations.
• rldjbpin 11mo ago
  feels like the main message is missed by seeing most of the discourse here: > Outages like today's are a good thing because they're a warning. They can force redundancy and resilience into systems. the advice is not to shun big companies and providers, but rather have a backup solution built-in for situations like this. switching solely to an in-house alternative is not always a great idea, but it can be a great backup solution.
• chris_wray 11mo ago
  I really wish we could build a truly decentralized server platform.
• a_state_full 11mo ago
  [dead]