28 ms·

Turns out half the internet has a single-point-of-failure called “Cloudflare”

972 points by StuntPope 6y ago

badRNG 6y ago
Interesting perspective, but it seems like this is just an ad for easyDNS and their "Proactive Nameservers," though I couldn't imagine a better time than the misstep of a behemoth of a competitor in this space. Not to detract from the more important discussion about the internet's dependence on Cloudflare overall.
- nvahalik 6y ago
  Hey, never let a competitors misfortune (misstep?) go to waste!
- sradman 6y ago
  It may be just an ad for easyDNS' Proactive Nameservers [1] product but it provides a roadmap for one possible solution to this type of problem. From a quick reading of the marketing info, the solution can be summarized as "Provision, Monitor, and Fail-Over DNS Name Servers across multiple DNS-as-a-Service providers". The question I have is whether the following constraint is artificially introduced or not: > We must be your domain registrar for this to work... IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem. The question comes down to build vs buy and whether the solution is general enough to warrant an Open Source Software solution. [1] https://easydns.com/dns/proactive-nameservers/ https://easydns.com/dns/proactive-nameservers/
  - Semaphor 6y ago
    From TFA: > The only requirement to use Proactive Nameservers is that we have to be your registrar, because we need to connect to the registry to update your nameserver delegation. So I guess technically this could be achieved with an API for your domain settings.
  - chronid 6y ago
    > IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem. The classic way of doing this is AXFR (your own DNS server is a "hidden master" and the DNS providers are the slaves). The problem is you won't be able to have redundancy at the registrar level, but that has historically at least been less of an issue.
  - pas 6y ago
    They want to be the registrar to be able to update your NS records. But ... that's not really important nor needed (So the answer to your question yes, it's likely artificial). Just use two anycast-ed IPs/domains. (Like Cloudflare.) The magic happens at BGP level. I considered CF as a domain registrar, but they don't allow setting the NS records. So you must use them. (They basically use sane no-nonsense domain registration as a way to gain leads for their main product. Pretty smart actually, because it's a great high-level add-on for their main product, but they just went ahead and made that the bait for everyone.) Anyway, ideally, if you add 2 separate sets of NS servers to your NS records then you eliminated this SPoF, great. Sure, it's your job to keep them updated, and in sync (preferably, to avoid problems like half of your users landing on a different CNAME/IP/etc). And recursive nameservers will handle the failover.
    - 1996 6y ago
      > Just use two anycast-ed IPs/domains What are the brands offering DNS at a flat rate over anycasted IP over a few continents? Most of the offers I see are per query at high rates. Setting up my own ASN to do this would be too expansive in IPv4
      - pas 6y ago
        I meant that easyDNS should handle the BGP for its clients, without requiring their clients to use them as registrars. There's HE.net's free DNS, and though they don't explicitly advertise as, it's anycasted. (Check via https://tools.keycdn.com/ping https://tools.keycdn.com/ping , try 216.66.80.18 [ns5.he.net].) https://www.cloudns.net/premium/ https://www.cloudns.net/premium/ seems to be quite affordable with no query limits :o
    - StuntPope 6y ago
      easyDNS has to be the registrar because only your registrar can change your nameserver delegation with the registry. This is, in essence, the registrar's job. To maintain your domain record and info, including nameserver delegation, with the registry. You could do it with BGP, but it is non-trivial and you need your own ASN to do that.
- donmcronald 6y ago
  > Proactive Nameservers is a patent-pending system that optimizes the nameserver delegation for your mission critical domain names. That's a huge negative and I can't believe they think it's a good marketing point. I don't want a patent encumbered, non-standard solution for critical infrastructure. > We must be your domain registrar for this to work. So they're updating the domain record at the registry level to facilitate failover? That's the only scenario I can think of where they _need_ to be your registrar. Assuming that's the case... I've always seen 24-48 hours quoted as the worst case wait when updating nameservers at the registry. I've never seen an explanation of how it works, what's allowed to be cached, how long it actually takes to update, etc.. How do they do it in a way that's suitable for failover? Do they have a special SLA with registries? How would the registries handle a deluge of nameserver updates? Imagine a Cloudflare scale failure and corresponding registry updates. Would the registry servers be able to handle it? I'd love to see a technical explanation of how their proactive nameservers system works.
dentemple 6y ago
If more companies are willing to provide the same level of service and price as Cloudflare, then they can get in on the game, too.
raverbashing 6y ago
If only DDOS attacks were taken seriously and their perpetrators punished accordingly (and maybe if the network had better ways of self-defense) instead of companies and websites having to fend for themselves (or having to resort to solutions like Cloudflare).
- ericlewis 6y ago
  I am not sure I understand this comment, in the context- cloudflare misconfigured some routes and it was quickly resolved. was this a DDoS?
  - jgrahamc 6y ago
    It was not.
  - t_sawyer 6y ago
    His point is why the internet has a single point of failure -> Cloudflare. They offer DDoS protection for free.
  - jaywalk 6y ago
    I think the point was that so many companies wouldn't have to rely on anti-DDoS protection from Cloudflare.
tyingq 6y ago
It's interesting to me that Cloudflare doesn't really have any competition with a similar business model. I suppose the free plan requires quite a lot of spending before the upgrades offset it.
- nerdponx 6y ago
  I don't need a free tier. I just need a "basic bitch" tier for my personal usage. Who are some Cloudflare alternatives for this?
  - corford 6y ago
    Depends on what you want but a very solid free CDN and DNS option is: host your site on netlify and use dns.he.net for your nameservers. Another good DNS option is dnsimple.com or, indeed, EasyDNS. For even more redundancy, use one provider as your domain registrar and another for your nameservers (and set short TTLs for your zones so you can re-point IPs quickly if you need to). For the other things Cloudflare offers on their free tier, I'm not sure what good alternatives exist (there must be some, I'm just not familiar with them outside of the obvious AWS alternatives). Edit: one caveat with above advice, I have no idea if netlify use cloudflare behind the scenes... Edit 2: For other options, checkout https://www.cdnperf.com/ https://www.cdnperf.com/ and https://www.dnsperf.com/ https://www.dnsperf.com/
  - KirinDave 6y ago
    If all you're doing is hosting and CDNing static content, the US cloud providers can do this very, very cheaply.
- kijin 6y ago
  There are other large CDNs like Akamai. They just don't compete with Cloudflare in the consumer sector. It probably doesn't matter for them because enterprise contracts are where all the money is at.
  - deleted 6y ago
    [deleted]
OutsmartDan 6y ago
Is it realistic for a small-medium sized business to have more than one DNS provider?
- throw0101a 6y ago
  > Is it realistic for a small-medium sized business to have more than one DNS provider? Yes: as the weblog post points out, you can have EasyDNS as your master with their multiple DNS servers, and then also have (e.g.) Route53 slaved to EasyDNS and have those in addition to EasyDNS in your records. DNS servers have had replication for decades.
- sukilot 6y ago
  It's easy enough if you sign up with a DNS provider provider.
deleted 6y ago
[deleted]
aneutron 6y ago
Okay here's the thing. I'm okay with people bashing other (competing) companies when they do wrong. However, I believe it is somewhat childish and uncalled for to bash another company, because of a mistake. First of all "I use easyDNS so I didn't notice it at all tbh" is not only a childish assertion, it's borderline a falsehood. You DO NOT offer the same services, nor the same scale. (No, VOD would not work if your VOD provider used Cloudflare's offering.) Second of all, as some have noted in other comments, you are very welcome to get just as big as them if you can offer similar (excellent) service and similar extremely competitive pricing. Otherwise, keep working on your offer and stop going for low hanging fruit like bashing the competitor for an outage when they literally might handle 1000x your traffic, and perhaps offer 20x the services your offer. Just a little rant ...
- bszupnick 6y ago
  Honestly I didn't notice the domain name, and I actually thought the author was being quite understanding by saying things like "This is inevitable and unavoidable and entirely excusable. Everybody blows up, every DNS provider in existence will experience downtime. No exceptions." and that they use Cloudflare themselves. This is obviously subjective, but to me it didn't come across as "they suck use us" but rather pointing out the inherent flaws in this quite popular SPOF and cautioning to avoid it.
  - techslave 6y ago
    the tone may have been moderate, but the message is smug. otoh it’s just marketing, and CF is no stranger to it, so i think it’s fair.
  - csharptwdec19 6y ago
    I think the root cause (which, IMO, you correctly point out) is lost on many modern developers. For whatever reason there's this modern idea that if a company A is paying money to company B for a service, that company B will handle all the 'hard stuff' for them. The end result is we have a lot of applications/infra built with SPOFs, in some cases known, but in many, swept under the rug and abstracted away to passing the buck in case of a large failure (i.e. major AWS/Cloudflare/Azure outages). You also see this at times when vendors pitch internal software solutions. I've been at more than one shop where a vendor's 'silver bullet' turned into a SPOF time-bomb because nobody considered this company's solution could fail. After all, the sales presentation said it had %nines%!
T3RMINATED 6y ago
If only DDOS attacks were taken seriously and their perpetrators punished accordingly (and maybe if the network had better ways of self-defense) instead of companies and websites having to fend for themselves.
drawkbox 6y ago
That is the problem with massive centralization even if it is market level and internally Cloudflare (or any other big fish) does decentralization/fail-over of their own. Many of these companies should have had fail-over to competitors at least for reliability. The problem with near market monopolization, oligopoly, even the singularity, the fail-case is catastrophic and may even wipe out decentralized, diffused, dispersed, decoupled system solutions that can't make it due to so much relative size from the big fish that it squashes them along the way. The bigger the ship the longer it takes to turn. This Cloudflare issue is like the recent Facebook SDK startup crashes where everyone has a single point of failure on Facebook SDK where people should be using or able to use the OpenGraph API directly as they need which is more robust to the app that uses it, it won't crash on startup. In business it is a goal to centralize to grow, in nature and robust systems it is more differentiation and decentralization to survive. There will always be a push and pull between these two forces. Systems and markets are like gardens. The garden must be maintained, new seeds planted and helped to grow from small to mid-sized, mid-sized plants the bulk of the garden, and then the larger plants need to be culled back when they get too big to not take the mid-sized and then all the resources from the new seeds/small plants. The problem is we have allowed the top end to take over the garden and when they fail they fail spectacularly. The bigger the scale the bigger they can fail.
- saagarjha 6y ago
  > Many of these companies should have had fail-over to competitors at least for reliability. How would you even set such a thing up? I fear that you might get a couple of collusionary companies that bail each other out and smaller providers might just be left out to dry…
  - Qub3d 6y ago
    The article posted actually talks about EasyDNS's implementation of exactly this. > At easyDNS we experienced so much pain from this reality that we created a system to automate flipping DNS providers at the first sign of trouble. > We call it Proactive Nameservers, and we’re the only company in the world doing it for some reason. Maybe this is because in order to provide a service like nameserver failover, it means a company has to admit to its customers the reality that their own nameservers may at some point, fail.
nickreese 6y ago
This is just an advertisement for easydns.
- lopis 6y ago
  Bingo. Still a good read, and easydns is just trying to profit over a screw-up from a competitor, but still essentially an ad.
  - toomuchtodo 6y ago
    > but still essentially an ad. Compared to the endless content marketing Cloudflare posts [1]? It's an ad, but they're still right. That's just good content marketing (informative, relevant, and perhaps you buy something because of it). [1] https://news.ycombinator.com/from?site=cloudflare.com https://news.ycombinator.com/from?site=cloudflare.com
- superkuh 6y ago
  The more people that use anything other than Cloudflare, the better. I had this conversation with people back in ~2010 about Facebook and they all ignored it. They will again, but this time the consequences of centralization will be even worse. It won't just be one single website that goes shitty with blockages and manipulation and censorship. It won't even be just the web. When Cloudflare achieves their goal of deep packet inspection at every peering and transit point it'll be the end of the internet as we knew it and the slow transition to just another cut apart "China-net (tm)".
deleted 6y ago
[deleted]
yokaze 6y ago
I am not sure, if I would trust anyone who misuses the term "Single-Point-of-Failure" on matters of reliability.
- yjftsjthsd-h 6y ago
  Why not? It's a single thing, that if it fails, causes your app/website/whatever to fail.
  - yokaze 6y ago
    Because it isn't a single thing, it is a redundant system. Redundant systems can also fail, that doesn't make it a _single_ point of failure. You can add another system in parallel as the vendor of the product suggests, or you can improve the resilience in the redundant system. To make a hyperbole: my galera cluster is failing, its a single point of failure, so I setup a cockroach cluster in parallel. In a way, it is right, as there are failure modes specific to the individual systems, but I think, it is incorrect, to label that a SpoF.
    - yjftsjthsd-h 6y ago
      That is a fair point; SPoF depends on what level you're looking at. A RAID array removes the SPoF that is a single disk, but still leaves a SPoF in the RAID controller or CPU or power supply; a ceph cluster can withstand the loss of a whole rack but could still fall to certain software bugs. Likewise, "cloud" companies are internally redundant, right up to the point where they aren't. It depends on how you scope the question.
arkitaip 6y ago
> We call it Proactive Nameservers, and we’re the only company in the world doing it for some reason. Wait, why? [0]: > Proactive Nameservers is a patent-pending system that optimizes the nameserver delegation for your mission critical domain names. Oh. [0] https://easydns.com/dns/proactive-nameservers/ https://easydns.com/dns/proactive-nameservers/
xwdv 6y ago
Turns out no one got fired for choosing Cloudflare.
ehsankia 6y ago
Didn't people also say the same thing about AWS a while back when that had a downtime? I guess the internet has multiple "Single-Point-Of-Failure"s.
- benbristow 6y ago
  Ironic since the internet (ARPANET) was specifically designed to not have a single point of failure
  - KirinDave 6y ago
    Doubly ironic since in doing this, they created a system where a protocol is the SPoF; namely nonsensical or false BGP advertisements can quickly kill the internet as a whole if done correctly.
- KirinDave 6y ago
  That's not actually a contradiction. When an individual website is considered as a system, it will have multiple points where the failure of a component inhibits the system. It's possible to have both cloudflare and your database as "SPoFs" and that "single" is not meant to imply everyone only gets one. It's absolutely true that if us-east-1 in AWS has a bad day, a significant fraction of the American digital economy will shut down. For some companies, the same is true of Azure and Google's various comparable offerings. I read your post as skeptical. Why would you be skeptical? If you care about keeping your product up, you absolutely should have a fallback for cloudflare if you're a customer of theirs. Now, you might not care (and actually, for most folks I submit you need not care), but the folks making sure Ambulances get timely push notifications and realtime driving instructions probably care quite a bit.
- slazaro 6y ago
  A "single point of failure" doesn't mean there's only one of them. It means the whole fails when the SPOF fails. But you can have many of them. A steel chain made of links has as many SPOF as links.
- whinybastard 6y ago
synaesthesisx 6y ago
What's funny about this outage is I'm sure many of of us (myself included) used this window to analyze large services and determine an increase in major Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced issues due to the Cloudflare outage! The situation has exposed just how critical Cloudflare is. I went ahead and bought calls ahead of NET earnings next month. Cloudflare is becoming an increasingly bigger part of the internet backbone. Purely speculating here, but I wouldn't put it past AWS or another large player acquiring them soon.
- giancarlostoro 6y ago
  > I wouldn't put it past AWS or another large player acquiring them soon. I really hope it doesn't come to this sadly. I'd be okay with DO or somebody who isn't as massive doing a merger. Maybe they can pull resources to make each other even successful and maintaining reasonable independence.
- dharmab 6y ago
  > an increase in major Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced issues due to the Cloudflare outage! Careful about this methodology. Some services at my org were impacted despite not being direct CloudFlare customers. They had external dependencies that used CloudFlare.
  - Fiveplus 6y ago
    So it's much bigger proverbial 'blast-radius' lest something happen to CloudFlare? Can you elaborate a bit on that part? I'm interested in knowing more.
    - dharmab 6y ago
      Simple case of dependencies failing. Not much to elaborate. e.g. NPM.js uses CloudFlare DNS, so services which needed to talk to NPM.js weren't able to do so.
- iamnothere 6y ago
JohnTHaller 6y ago
Many of us don't even know it. My DNS for PortableApps.com is run through Digital Ocean, which uses Cloudflare for DNS.
- gripfx 6y ago
  On an unrelated note. Thank you so much for PortableApps.com! It was invaluable at university when studying in the library. To this day, I still use it for utilities that don't need to be installed on my desktop.
  - mrsalt 6y ago
    I am also grateful for PortableApps.com. Along with Scoop and Homebrew, they make using a system without root or admin privileges a really nice experience, in both Windows and Linux. My sincere thanks to John and all the PortableApps.com contributors.
    - JohnTHaller 6y ago
      You're welcome! I'm glad it's helping you be more productive!
  - JohnTHaller 6y ago
    You're welcome, I'm glad it's helped and helping you! I try to keep it growing and relevant with a more synced cloud folder/work or school laptop/keep work and personal separate bent these days.
michaelbuckbee 6y ago
Honest question: I've never really understood the back of the napkin math of how Cloudflare functions economically, which I feel would go a long way towards my understanding of why/how they were able to become such an integral and generally positive part of the Internet. Did they have some crazy in to get cheap bandwidth? Did they bet big on bandwidth prices falling? Did they figure something else out that nobody saw? Do they just to a tremendous job of migrating sites from free to paid plans?
- jgrahamc 6y ago
  Read our S-1, it's all in there. https://www.sec.gov/Archives/edgar/data/1477333/000119312519222176/d735023ds1.htm https://www.sec.gov/Archives/edgar/data/1477333/000119312519...
  - potency 6y ago
    Off topic, but any regrets on getting involved in content policing? It always sat with me as wrong and a disturbing precedent that an internet backbone service such as yourselves would make it their business to shut down unsavory yet legal speech.
    - nix23 6y ago
      Wikileaks and Piratebay are customers of CF, so your 'legal speech' must be really something. Any links?
      - largbae 6y ago
        Maybe GP is referring to 1.1.1.2 and 1.1.1.3, the DNS resolvers that filter malware and malware+adult content respectively? Both are optional alternatives to the unfiltered 1.1.1.1 DNS service if so...
      - 236dev 6y ago
        The only ones I can think of are The Daily Stormer and 8chan
eloff 6y ago
Now that Cloudflare is also a registrar, they could pretty easily implement a nameserver failover like EasyDNS. I hope this event underscores the importance of that to them. It's worth noting Cloudflare also supports secondary DNS, but only for enterprise customers: https://blog.cloudflare.com/secondary-dns-a-faster-more-resilient-way-to-serve-your-dns-records/ https://blog.cloudflare.com/secondary-dns-a-faster-more-resi...
psim1 6y ago
What responsibility and reparative measures has Cloudflare taken for Friday's incident? Was anyone fired for the mistake?
- sujinge9 6y ago
  Why would firing someone make you feel like reparative measures have been taken?
  - psim1 6y ago
    Reparations would show an actual sense of responsibility. Firing someone would be appropriate if they were negligent. Other measures might be more appropriate. Is it enough that any time there's a Cloudflare incident, all we get are lengthy blog posts and sorries from Cloudflare?
    - gizmo385 6y ago
      Firing people for making mistakes is just going to foster a culture of secrecy and shame. Being so quick to fire is not how you retain talent and it isn't how you foster a healthy, blameless development culture in your workplace.
    - axaxs 6y ago
      Firing people for making mistakes is a great way to watch productivity dive. The easiest way to prevent mistakes is to do nothing at all.
      - psim1 6y ago
        I understand the point being made here, but what are those affected supposed to take away? Cloudflare made a mistake that caused (x millions of dollars of lost online commerce revenues, y number of missed telehealth sessions, etc.) and since we do not punish mistakes, nothing was done. Sorry everyone!
        6y ago
nonbirithm 6y ago
Question: is it even possible to have DDoS protection without using a provider of it which becomes a single point of failure? Or is it maybe possible to decouple this single feature from everything else that Cloudflare provides that could take out all the sites in the future from an unrelated misconfiguration? I don't see the centralization as a positive, but I'm wondering what percentage of the websites that were taken offline see themselves as having no choice but to use Cloudflare in order to prevent themselves from being taken down anyway from malicious actors instead of by accident.
- deleted 6y ago
  [deleted]
- divbzero 6y ago
  I think you could use Cloudflare as your primary DNS provider and benefit from their DDoS protection, but also specify backup DNS name servers with a different DNS provider in case Cloudflare fails.
- sschueller 6y ago
  Sure, Arbour and others sell devices to deal with ddos and many isps have clusters of these which you can use. Of course this is a service that costs money.
tannhaeuser 6y ago
Whether this is an ad piece by a competitor or not, the problem with monopolies is that "the market" (if there is one) gets skewed incentives. Cloudflare has received heavy investment by FAANG (+MS) [1] before their IPO, so rather than eg Google or others with a vested interest and capability stepping up the game and invest into new IP control plane-level DDOS protection standards or similar, the situation smells more like a backdoor deal, such as an agreement to not go after a particular market segment. Let's also not forget Cloudflare in particular have been accused to host/hide the very bad boys that make protection from DDOS necessary in the first place. Whether or not that is the case, a quasi-monopoly leaves customers with no choice. [1]: https://petri.com/microsoft-google-and-others-invest-in-cloudflare https://petri.com/microsoft-google-and-others-invest-in-clou...
TedDoesntTalk 6y ago
> Turns out half the internet has a Single-Point-of-Failure called “Cloudflare” The other one is called AWS.
- louwrentius 6y ago
  So in the end, who cares about single point of failures?
  - timbit42 6y ago
    Militaries. The internet was created so the US President could command the US military even in the case of nuclear war knocking out many of the internet's nodes.
solotronics 6y ago
Maybe it would make sense to have multiple independent sections of backbone at the BGP level. Instead of having one public AS/backbone, break it down into regions at least so that it is more confederated.
citizenpaul 6y ago
I love all the comments about fail-over for DDOS/DNS protection. What is your budget? Well we are looking at around $0.00 for our maximum allowance. Ok so single point of failure it is I guess we are done here. Companies only say they care when there is a problem, the reality is that they dont.
- divbzero 6y ago
  It obviously takes more than $0.00 but not that much more. It’s a matter of adding a second DNS provider and making sure you replicate DNS records manually or with AFXR. What’s really puzzling is if the same companies spend money on active/passive failover for application and database servers while overlooking DNS single point of failure.
- FireBeyond 6y ago
  Precisely. I remember when the CEO of my old company came to me and said (with respect to moving from a on-prem model to SaaS), "What's our SLA going to be?" "Well, what do you want it to be?" Give me a number and I'll tell you how much it will cost and how long it will take to get there.
crazygringo 6y ago
I guess this is off-topic, but the stock photo they're using is cracking me up. Guy at work... coffee cup on a tablet he's using for a coaster... except he's also drinking whiskey from a beautiful crystal glass... there's a folded paper airplane... there's just so much to unpack here, it's pretty hilarious.
- hinkley 6y ago
  > except he's also drinking whiskey from a beautiful crystal glass And he has left the stopper off of the decanter like some sort of animal. I think we are supposed to believe that the person here was just dicking around online, fiddling with paper, finishing his morning coffee, when all of a sudden he gets an email asking if anyone knows what's going on with the website. So he stops what he was (not) doing, puts down his coffee, and starts poking around. At which point he realizes he needs something stronger than coffee. As he is pouring his glass he is confronted with the true horror of the situation, drops the stopper on the floor and just holds his face wondering why the Universe hates him. I wonder if we can get JJ Abrams to option the movie rights.
niutech 6y ago
The solution is the Decentralized Web (DWeb), such as IPFS (https://ipfs.io https://ipfs.io), Freenet (https://freenetproject.org https://freenetproject.org), GNUnet (https://gnunet.org https://gnunet.org) or Hypercore (https://hypercore-protocol.org https://hypercore-protocol.org). We should start using them to avoid centralization and embrace freedom.
- nemothekid 6y ago
  Can you explain a bit more how this is a solution? Cloudflare isn't facebook. They have plenty of competitors, they don't have a massive moat, and they are almost exclusively used by businesses. Despite all of this, we should ask ourselves how we even got here. Why would companies move to DWeb, when they are already choosing to use Cloudflare instead of Fastly/Cloudfront/Akamai.
  - return1 6y ago
    No salespeople to sell dweb
- mattl 6y ago
  How do I publish a website on one of these?
  - yjftsjthsd-h 6y ago
    https://www.dgendill.com/posts/technology/2019-10-15-publish-website-to-ipfs.html https://www.dgendill.com/posts/technology/2019-10-15-publish...
- cortesoft 6y ago
  Can any of those handle massive scale?
surround 6y ago
Instead of trusting any DNS provider with your queries, I recommend running your own recursive resolver with Unbound. https://nlnetlabs.nl/projects/unbound/about/ https://nlnetlabs.nl/projects/unbound/about/
- everfree 6y ago
  But then aren't you just trusting your VPS provider or ISP instead?
  - surround 6y ago
    Unfortunately, ISPs can see the domains you connect to, even if you use DNS-over-HTTPS. https://irtf.org/anrw/2019/slides-anrw19-final44.pdf https://irtf.org/anrw/2019/slides-anrw19-final44.pdf
actuator 6y ago
Does anyone know a dashboard/list for past Akamai outages. Surprisingly, I haven't seen a lot of news about Akamai downtimes. I searched on Google and the last one I found in a news report is from 2011 when its customers like Facebook, Twitter were impacted. They were a PITA to work with when I used them in the past but if they are really that good in service availability, you can have some justification for their overpriced service.
- EE84M3i 6y ago
  AFAIK Akamai only makes their service notifications available directly to subscribers.
louwrentius 6y ago
Who at a C-level position is going to tell anybody that the potential risk of Cloudflare (or Amazon/Azure/GCP) going down should be protected against? I would applaud them, but I wonder.
- divbzero 6y ago
  There’s a lot of truth to this. Cloudflare for now has achieved IBM status in that no one will be fired for choosing Cloudflare, in spite of any issues that arise.
- Nasrudith 6y ago
  My guess is Wall Street HFT or other financial areas with very strict unscheduled downtime penalties where they are effectively incentivised to be batshit paranoid as it would take decades for any penny pinching to remotely pay off. I don't know if many of them use Cloudflare for their domains though.
johnghanks 6y ago
lmfao this is literally a hit piece from a competitor.
jonplackett 6y ago
Similar problems are happening with crypto - yeah it’s distributed but so many people are using Coinbase that if they go down it’s going to cause a lot of problems
zelphirkalt 6y ago
Well, I usually block it anyway, because it is not only a single point of failure, but also a single point of concentration of data, that can be used to track, spy on and profile users. As such, I do not blindly trust Cloudflare. I remain sceptical, no matter how positive their public image is. Especially I would not set my DNS to cloudflare.
- ampersandy 6y ago
  Who do you use for DNS then that you do trust?
  - yjftsjthsd-h 6y ago
    For lookups, it's not that hard to do your own recursive resolver.
mountainboy 6y ago
More importantly, CloudFlare is the world's largest man-in-the-middle. You think your TLS connection is e2e between you and the website you are visiting? Not so much... because the website has given their certs to CF. Worse, sometimes/often the connection between CF and the website is not encrypted at all.
neycoda 6y ago
When Cloudflare causes the problem they were built to solve.
deleted 6y ago
[deleted]
black_puppydog 6y ago
What I find really shocking is the abundant use of CF on piracy websites of all things. Not the serious ones of course, SciHub and library genesis are mirrored differently. But a lot of small torrent websites and such simply won't load without JS and specifically CF code. It's pretty crazy. Luckily I don't use any of those bEcAuSe IlLeGaL but still, I find it really depressing, especially when webtorrent, IPFS etc are available, and frankly many of those pages will never have to bear a load that makes CF a requirement.
- jdc0589 6y ago
  its not about caching or handling normal traffic. its about ddos protection. sites like that are frequent targets.
  - black_puppydog 6y ago
    and there should be enough of them that that shouldn't matter.
phkahler 6y ago
If you keep much of a website simple - plain html and css - won't that reduce the need for a CDN in the first place?
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
- foxfired 6y ago
  Yes, but cloudflare reach is much deeper then that. I am not a cloudflare customer but my all websites failed that day. The reason was digitalocean uses cloudflare, I use digitalocean. So apparently I depend on cloudflare.
hinkley 6y ago
High availability is an insurance game, and perhaps we need to start treating it that way. Rather than admitting that your customers need to maintain a business relationship with your competitors, you need to admit you need to maintain a business relationship with your competitors. That we need a moral equivalent of underwriting in the cloud space.
tuxninja 6y ago
AWS in 2012, DynDNS after that, now Cloudflare...I wrote about this a few years ago, the threat of the singularity of the Internet. What was distributed will be centralized again. http://tuxlabs.com/?p=430 http://tuxlabs.com/?p=430
johnklos 6y ago
I think people are Cloudflare fans simply because so many other services suck more. I gave Cloudflare a fair shake. But after hearing their lies way too many times, I'm calling them out for being deceptive and unscrupulous. After being told that sites pretending to host Adobe Flash updaters and pretending to be Bank of America can't be taken down by Cloudflare because of their rights to free speech, I knew their attempts to pretend to be one of us, attempts to pretend to care, were nothing but bullshit. They claim they don't host. If hosting DNS is not hosting, then what do you call DNS hosting? You literally CANNOT use their abuse web form to report domains which use Cloudflare just for hosting DNS. They do not handle abuse sent to their abuse email address (they simply send a form response saying to spend ten minutes filling out their crappy form that has all sorts of problems). Of course, their web proxy services are also "not hosting", even though they're protecting all sorts of scammers. So why should we think they're not bullshitting us when they run 1.1.1.1 and tell us they're not logging? Why should we trust them more than our ISPs by running DoH through them? They WANT us to be dependent on them, because the more control they have, the more money they can make. It's dangerous, and they've shown they have no honor. I've genuinely tried to correspond with them on Twitter, and they excel at not answering the question asked but instead just diverting. It's scummy, unprofessional behavior and I encourage everyone to consider whether they deserve anyone's data or business.
- james412 6y ago
  > They WANT us to be dependent on them I think this is an important point about CloudFlare that can't be made often enough. It's been some years since I first noticed, and it seems as true today as it was then: wedging themselves into core Internet services and data flows seems to be an intentional part of CloudFlare's strategy. There is no case given the architecture of the Internet where one company need be exposed to so many traffic flows from millions of people. The search engines got there first and we eventually stopped complaining, but this does not make it any more justifiable to copy the model. What reason would a company have for desiring this outcome? We know Google can detect and predict flu outbreaks. Imagine what is possible when you have every click on every target web site. There is a fair chance their data is already approaching the comprehensiveness of Google, and I'd be surprised, if not disappointed to learn they were not already working on unannounced (now or eternally) internal intelligence products based on that data. There are simply too many pockets who would be willing to pay for it.
ivanvanderbyl 6y ago
Pretty poor form calling out a competitor for something like this. Not the first time a DNS provider has done this, and won't be the last.
knorker 6y ago
The other half is on AWS?
miki123211 6y ago
Cloudflare is horrible for blind people. Screen readers, the programs that use synthesized speech to tell us what's on the screen, cannot read images. Good captchas usually have audio equivalents (which come with their own set of problems), but this one doesn't. If you're blind and flagged by Cloudflare for some reason, you're cut off from accessing half the internet, potentially critical banking/governmental/medical/communications/educational services. We rely on the internet way more than our sighted peers, so this is very important. This has recently happened to me on a few sites, fortunately not critical ones, but it was not a pleasant experience nonetheless. CF engineers, please fix this ASAP. I'm surprised there still isn't a huge lawsuit over this, as this is clearly violating all sorts of laws.
- deleted 6y ago
  [deleted]
- strombofulous 6y ago
  I use audio captchas. Google will usually only let you do 2 or 3 before banning you and making you do image-based ones. I'm pretty sure the button is just there to make it seem accessible.
  - dependenttypes 6y ago
    I had exactly the same issue. I am surprised how neither cloudflare nor google have been sued for making most of the web inaccessible to people with disabilities.
    - lopmotr 6y ago
      Shouldn't that liability be with the operators of websites that use those captchas if they're required to be blind-accessible? If they get sued, it'll apply pressure up the supply chain.
      - kbenson 6y ago
        I agree. You don't sue the contractor because they built to the specification you provided. At the same time, if Google is advertising Recaptcha as accessible and it's not really, then they need to be held accountable for that, because that has a real impact on huge swaths of the internet, and especially for sites that try to do the right thing and find out Google has screwed them.
valuearb 6y ago
“Cloudflare apparently fat-fingered a routing update and sent all of their global traffic to a single POP, vaporizing it almost instantly.” Made me chuckle, as it gave me the image of a large server in some massive server farm glowing red, then bursting in a massive burst of light as dozens of bearded Sysadmins run out of the building screaming.
- sciurus 6y ago
  In my experience the sysadmins would be running in the opposite direction.
  - Baeocystin 6y ago
    As usual, a relevant one: https://xkcd.com/705/ https://xkcd.com/705/
Uhhrrr 6y ago
> But if you want to use a preferred DNS provider, such as Cloudflare, who use their DNS responses to optimize your website proxy. That works best most of the time, so then you want to go with an active/passive model that will step back when things are going according to plan, and then when these periodic network cataclysms do occur (and they will), they step into the breach and update your nameservers so that you at least stay up until the crisis is over. Copy editors are cheap and your reputation shouldn't be.
WarOnPrivacy 6y ago
This is the 3rd DNS related outage I've seen in a week. Frountier Communications lost their DNS. Trying to recall what the 3rd one was. My Unbound resolver round-robins DNS-over-TLS requests, between Cloudflare & Quad9. Cloudflare's outage never impacted us that I could tell. Nevertheless, I am reminded that I ought to add a couple of DoT providers (who aren't Google). Not sure who else came online since I setup.
anthk 6y ago
Cloudflare doesn't work without JS, even for static jsless sites. It's a pest.
aabbcc1241 6y ago
There are some (research) work on proof-of-human protocol, interesting direction
aww_dang 6y ago
Cloudflare is hell when you need to load scripts and css 403'd by them. Here's my work around: 1) open developer tools 2) refresh page 3) move tab into a new window 4) find a blocked resource in the network tab of developer tools and open it in a new tab 5) verify humanity 6) repeat step 4 and sometimes 5 for each resource the page requires 7) move the problematic site out of the new window and close all of these tabs at once It is a terrible experience. CF devs and publishers have no idea how inconvenient their service is. I wonder if they have ever lived in a region where every ISP is both incompetent and listed in the CBL? I frequently skip sites that use CF. The captchas are obnoxious. The entire concept of a webapp firewall seems a bit backwards to me. Developers should fix their insecure applications.
tonyfader 6y ago
you can warn people, but they don't listen...
erichocean 6y ago
"Turns out over half the Internet has a single point of failure called BGP."
imvetri 6y ago
Haha
dschuetz 6y ago
When I was figuring out how DNS works back 10 years ago, I was told "Why don't you just use 8.8.8.8 everywhere? Why do you need your own DNS server for anyway?" every single time when I asked a very specific configuration question. Some years later 8.8.8.8 was just replaced with 1.1.1.1 with same critical counter questions instead of helping. So, it appears to me, instead of understanding DNS and routing, people and tech-bro businesses take shortcuts by using centralized infrastructure for their systems, creating dangerous configurations. Now, Cloudflare have made a mistake, and half the Internet crumbled down. See the irony?
satokema_work 6y ago
CHEAP V1AGRA N0W!!11
rchaud 6y ago
Starting yesterday, I have been getting Cloudflare's hCaptcha image identification quizzes on every site that uses them. Googling the issue indicates that the IP address range may have been blacklisted, if say my laptop (a Mac that doesn't visit any shady sites) or my router has been compromised and is now running a botnet or something. It seems unlikely because my phone can access sites on wifi fine, and again my Mac doesn't appear to have any malware. Could it be that the outage is somehow relate to Cloudflare putting these captchas up as a precautionary measure?

972 points by StuntPope 6y ago

Interesting perspective, but it seems like this is just an ad for easyDNS and their "Proactive Nameservers," though I couldn't imagine a better time than the misstep of a behemoth of a competitor in this space. Not to detract from the more important discussion about the internet's dependence on Cloudflare overall.

nvahalik 6y ago
Hey, never let a competitors misfortune (misstep?) go to waste!
sradman 6y ago
It may be just an ad for easyDNS' Proactive Nameservers [1] product but it provides a roadmap for one possible solution to this type of problem. From a quick reading of the marketing info, the solution can be summarized as "Provision, Monitor, and Fail-Over DNS Name Servers across multiple DNS-as-a-Service providers". The question I have is whether the following constraint is artificially introduced or not: > We must be your domain registrar for this to work... IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem. The question comes down to build vs buy and whether the solution is general enough to warrant an Open Source Software solution. [1] https://easydns.com/dns/proactive-nameservers/ https://easydns.com/dns/proactive-nameservers/
- Semaphor 6y ago
  From TFA: > The only requirement to use Proactive Nameservers is that we have to be your registrar, because we need to connect to the registry to update your nameserver delegation. So I guess technically this could be achieved with an API for your domain settings.
- chronid 6y ago
  > IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem. The classic way of doing this is AXFR (your own DNS server is a "hidden master" and the DNS providers are the slaves). The problem is you won't be able to have redundancy at the registrar level, but that has historically at least been less of an issue.
- pas 6y ago
  They want to be the registrar to be able to update your NS records. But ... that's not really important nor needed (So the answer to your question yes, it's likely artificial). Just use two anycast-ed IPs/domains. (Like Cloudflare.) The magic happens at BGP level. I considered CF as a domain registrar, but they don't allow setting the NS records. So you must use them. (They basically use sane no-nonsense domain registration as a way to gain leads for their main product. Pretty smart actually, because it's a great high-level add-on for their main product, but they just went ahead and made that the bait for everyone.) Anyway, ideally, if you add 2 separate sets of NS servers to your NS records then you eliminated this SPoF, great. Sure, it's your job to keep them updated, and in sync (preferably, to avoid problems like half of your users landing on a different CNAME/IP/etc). And recursive nameservers will handle the failover.
  - 1996 6y ago
    > Just use two anycast-ed IPs/domains What are the brands offering DNS at a flat rate over anycasted IP over a few continents? Most of the offers I see are per query at high rates. Setting up my own ASN to do this would be too expansive in IPv4
    - pas 6y ago
      I meant that easyDNS should handle the BGP for its clients, without requiring their clients to use them as registrars. There's HE.net's free DNS, and though they don't explicitly advertise as, it's anycasted. (Check via https://tools.keycdn.com/ping https://tools.keycdn.com/ping , try 216.66.80.18 [ns5.he.net].) https://www.cloudns.net/premium/ https://www.cloudns.net/premium/ seems to be quite affordable with no query limits :o
  - StuntPope 6y ago
    easyDNS has to be the registrar because only your registrar can change your nameserver delegation with the registry. This is, in essence, the registrar's job. To maintain your domain record and info, including nameserver delegation, with the registry. You could do it with BGP, but it is non-trivial and you need your own ASN to do that.
donmcronald 6y ago
> Proactive Nameservers is a patent-pending system that optimizes the nameserver delegation for your mission critical domain names. That's a huge negative and I can't believe they think it's a good marketing point. I don't want a patent encumbered, non-standard solution for critical infrastructure. > We must be your domain registrar for this to work. So they're updating the domain record at the registry level to facilitate failover? That's the only scenario I can think of where they _need_ to be your registrar. Assuming that's the case... I've always seen 24-48 hours quoted as the worst case wait when updating nameservers at the registry. I've never seen an explanation of how it works, what's allowed to be cached, how long it actually takes to update, etc.. How do they do it in a way that's suitable for failover? Do they have a special SLA with registries? How would the registries handle a deluge of nameserver updates? Imagine a Cloudflare scale failure and corresponding registry updates. Would the registry servers be able to handle it? I'd love to see a technical explanation of how their proactive nameservers system works.

dentemple 6y ago

If more companies are willing to provide the same level of service and price as Cloudflare, then they can get in on the game, too.

raverbashing 6y ago

If only DDOS attacks were taken seriously and their perpetrators punished accordingly (and maybe if the network had better ways of self-defense) instead of companies and websites having to fend for themselves (or having to resort to solutions like Cloudflare).

ericlewis 6y ago
I am not sure I understand this comment, in the context- cloudflare misconfigured some routes and it was quickly resolved. was this a DDoS?
- jgrahamc 6y ago
  It was not.
- t_sawyer 6y ago
  His point is why the internet has a single point of failure -> Cloudflare. They offer DDoS protection for free.
- jaywalk 6y ago
  I think the point was that so many companies wouldn't have to rely on anti-DDoS protection from Cloudflare.

tyingq 6y ago

It's interesting to me that Cloudflare doesn't really have any competition with a similar business model. I suppose the free plan requires quite a lot of spending before the upgrades offset it.

nerdponx 6y ago
I don't need a free tier. I just need a "basic bitch" tier for my personal usage. Who are some Cloudflare alternatives for this?
- corford 6y ago
  Depends on what you want but a very solid free CDN and DNS option is: host your site on netlify and use dns.he.net for your nameservers. Another good DNS option is dnsimple.com or, indeed, EasyDNS. For even more redundancy, use one provider as your domain registrar and another for your nameservers (and set short TTLs for your zones so you can re-point IPs quickly if you need to). For the other things Cloudflare offers on their free tier, I'm not sure what good alternatives exist (there must be some, I'm just not familiar with them outside of the obvious AWS alternatives). Edit: one caveat with above advice, I have no idea if netlify use cloudflare behind the scenes... Edit 2: For other options, checkout https://www.cdnperf.com/ https://www.cdnperf.com/ and https://www.dnsperf.com/ https://www.dnsperf.com/
- KirinDave 6y ago
  If all you're doing is hosting and CDNing static content, the US cloud providers can do this very, very cheaply.
kijin 6y ago
There are other large CDNs like Akamai. They just don't compete with Cloudflare in the consumer sector. It probably doesn't matter for them because enterprise contracts are where all the money is at.
- deleted 6y ago
  [deleted]

OutsmartDan 6y ago

Is it realistic for a small-medium sized business to have more than one DNS provider?

throw0101a 6y ago
> Is it realistic for a small-medium sized business to have more than one DNS provider? Yes: as the weblog post points out, you can have EasyDNS as your master with their multiple DNS servers, and then also have (e.g.) Route53 slaved to EasyDNS and have those in addition to EasyDNS in your records. DNS servers have had replication for decades.
sukilot 6y ago
It's easy enough if you sign up with a DNS provider provider.

deleted 6y ago

[deleted]

aneutron 6y ago

Okay here's the thing. I'm okay with people bashing other (competing) companies when they do wrong. However, I believe it is somewhat childish and uncalled for to bash another company, because of a mistake. First of all "I use easyDNS so I didn't notice it at all tbh" is not only a childish assertion, it's borderline a falsehood. You DO NOT offer the same services, nor the same scale. (No, VOD would not work if your VOD provider used Cloudflare's offering.) Second of all, as some have noted in other comments, you are very welcome to get just as big as them if you can offer similar (excellent) service and similar extremely competitive pricing. Otherwise, keep working on your offer and stop going for low hanging fruit like bashing the competitor for an outage when they literally might handle 1000x your traffic, and perhaps offer 20x the services your offer. Just a little rant ...

bszupnick 6y ago
Honestly I didn't notice the domain name, and I actually thought the author was being quite understanding by saying things like "This is inevitable and unavoidable and entirely excusable. Everybody blows up, every DNS provider in existence will experience downtime. No exceptions." and that they use Cloudflare themselves. This is obviously subjective, but to me it didn't come across as "they suck use us" but rather pointing out the inherent flaws in this quite popular SPOF and cautioning to avoid it.
- techslave 6y ago
  the tone may have been moderate, but the message is smug. otoh it’s just marketing, and CF is no stranger to it, so i think it’s fair.
- csharptwdec19 6y ago
  I think the root cause (which, IMO, you correctly point out) is lost on many modern developers. For whatever reason there's this modern idea that if a company A is paying money to company B for a service, that company B will handle all the 'hard stuff' for them. The end result is we have a lot of applications/infra built with SPOFs, in some cases known, but in many, swept under the rug and abstracted away to passing the buck in case of a large failure (i.e. major AWS/Cloudflare/Azure outages). You also see this at times when vendors pitch internal software solutions. I've been at more than one shop where a vendor's 'silver bullet' turned into a SPOF time-bomb because nobody considered this company's solution could fail. After all, the sales presentation said it had %nines%!

T3RMINATED 6y ago

drawkbox 6y ago

That is the problem with massive centralization even if it is market level and internally Cloudflare (or any other big fish) does decentralization/fail-over of their own. Many of these companies should have had fail-over to competitors at least for reliability. The problem with near market monopolization, oligopoly, even the singularity, the fail-case is catastrophic and may even wipe out decentralized, diffused, dispersed, decoupled system solutions that can't make it due to so much relative size from the big fish that it squashes them along the way. The bigger the ship the longer it takes to turn. This Cloudflare issue is like the recent Facebook SDK startup crashes where everyone has a single point of failure on Facebook SDK where people should be using or able to use the OpenGraph API directly as they need which is more robust to the app that uses it, it won't crash on startup. In business it is a goal to centralize to grow, in nature and robust systems it is more differentiation and decentralization to survive. There will always be a push and pull between these two forces. Systems and markets are like gardens. The garden must be maintained, new seeds planted and helped to grow from small to mid-sized, mid-sized plants the bulk of the garden, and then the larger plants need to be culled back when they get too big to not take the mid-sized and then all the resources from the new seeds/small plants. The problem is we have allowed the top end to take over the garden and when they fail they fail spectacularly. The bigger the scale the bigger they can fail.

saagarjha 6y ago
> Many of these companies should have had fail-over to competitors at least for reliability. How would you even set such a thing up? I fear that you might get a couple of collusionary companies that bail each other out and smaller providers might just be left out to dry…
- Qub3d 6y ago
  The article posted actually talks about EasyDNS's implementation of exactly this. > At easyDNS we experienced so much pain from this reality that we created a system to automate flipping DNS providers at the first sign of trouble. > We call it Proactive Nameservers, and we’re the only company in the world doing it for some reason. Maybe this is because in order to provide a service like nameserver failover, it means a company has to admit to its customers the reality that their own nameservers may at some point, fail.

nickreese 6y ago

This is just an advertisement for easydns.

lopis 6y ago
Bingo. Still a good read, and easydns is just trying to profit over a screw-up from a competitor, but still essentially an ad.
- toomuchtodo 6y ago
  > but still essentially an ad. Compared to the endless content marketing Cloudflare posts [1]? It's an ad, but they're still right. That's just good content marketing (informative, relevant, and perhaps you buy something because of it). [1] https://news.ycombinator.com/from?site=cloudflare.com https://news.ycombinator.com/from?site=cloudflare.com
superkuh 6y ago
The more people that use anything other than Cloudflare, the better. I had this conversation with people back in ~2010 about Facebook and they all ignored it. They will again, but this time the consequences of centralization will be even worse. It won't just be one single website that goes shitty with blockages and manipulation and censorship. It won't even be just the web. When Cloudflare achieves their goal of deep packet inspection at every peering and transit point it'll be the end of the internet as we knew it and the slow transition to just another cut apart "China-net (tm)".

deleted 6y ago

[deleted]

yokaze 6y ago

I am not sure, if I would trust anyone who misuses the term "Single-Point-of-Failure" on matters of reliability.

yjftsjthsd-h 6y ago
Why not? It's a single thing, that if it fails, causes your app/website/whatever to fail.
- yokaze 6y ago
  Because it isn't a single thing, it is a redundant system. Redundant systems can also fail, that doesn't make it a _single_ point of failure. You can add another system in parallel as the vendor of the product suggests, or you can improve the resilience in the redundant system. To make a hyperbole: my galera cluster is failing, its a single point of failure, so I setup a cockroach cluster in parallel. In a way, it is right, as there are failure modes specific to the individual systems, but I think, it is incorrect, to label that a SpoF.
  - yjftsjthsd-h 6y ago
    That is a fair point; SPoF depends on what level you're looking at. A RAID array removes the SPoF that is a single disk, but still leaves a SPoF in the RAID controller or CPU or power supply; a ceph cluster can withstand the loss of a whole rack but could still fall to certain software bugs. Likewise, "cloud" companies are internally redundant, right up to the point where they aren't. It depends on how you scope the question.

arkitaip 6y ago

> We call it Proactive Nameservers, and we’re the only company in the world doing it for some reason. Wait, why? [0]: > Proactive Nameservers is a patent-pending system that optimizes the nameserver delegation for your mission critical domain names. Oh. [0] https://easydns.com/dns/proactive-nameservers/ https://easydns.com/dns/proactive-nameservers/

xwdv 6y ago

Turns out no one got fired for choosing Cloudflare.

ehsankia 6y ago

Didn't people also say the same thing about AWS a while back when that had a downtime? I guess the internet has multiple "Single-Point-Of-Failure"s.

benbristow 6y ago
Ironic since the internet (ARPANET) was specifically designed to not have a single point of failure
- KirinDave 6y ago
  Doubly ironic since in doing this, they created a system where a protocol is the SPoF; namely nonsensical or false BGP advertisements can quickly kill the internet as a whole if done correctly.
KirinDave 6y ago
That's not actually a contradiction. When an individual website is considered as a system, it will have multiple points where the failure of a component inhibits the system. It's possible to have both cloudflare and your database as "SPoFs" and that "single" is not meant to imply everyone only gets one. It's absolutely true that if us-east-1 in AWS has a bad day, a significant fraction of the American digital economy will shut down. For some companies, the same is true of Azure and Google's various comparable offerings. I read your post as skeptical. Why would you be skeptical? If you care about keeping your product up, you absolutely should have a fallback for cloudflare if you're a customer of theirs. Now, you might not care (and actually, for most folks I submit you need not care), but the folks making sure Ambulances get timely push notifications and realtime driving instructions probably care quite a bit.
slazaro 6y ago
A "single point of failure" doesn't mean there's only one of them. It means the whole fails when the SPOF fails. But you can have many of them. A steel chain made of links has as many SPOF as links.
whinybastard 6y ago

synaesthesisx 6y ago

What's funny about this outage is I'm sure many of of us (myself included) used this window to analyze large services and determine an increase in major Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced issues due to the Cloudflare outage! The situation has exposed just how critical Cloudflare is. I went ahead and bought calls ahead of NET earnings next month. Cloudflare is becoming an increasingly bigger part of the internet backbone. Purely speculating here, but I wouldn't put it past AWS or another large player acquiring them soon.

giancarlostoro 6y ago
> I wouldn't put it past AWS or another large player acquiring them soon. I really hope it doesn't come to this sadly. I'd be okay with DO or somebody who isn't as massive doing a merger. Maybe they can pull resources to make each other even successful and maintaining reasonable independence.
dharmab 6y ago
> an increase in major Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced issues due to the Cloudflare outage! Careful about this methodology. Some services at my org were impacted despite not being direct CloudFlare customers. They had external dependencies that used CloudFlare.
- Fiveplus 6y ago
  So it's much bigger proverbial 'blast-radius' lest something happen to CloudFlare? Can you elaborate a bit on that part? I'm interested in knowing more.
  - dharmab 6y ago
    Simple case of dependencies failing. Not much to elaborate. e.g. NPM.js uses CloudFlare DNS, so services which needed to talk to NPM.js weren't able to do so.
iamnothere 6y ago

JohnTHaller 6y ago

Many of us don't even know it. My DNS for PortableApps.com is run through Digital Ocean, which uses Cloudflare for DNS.

gripfx 6y ago
On an unrelated note. Thank you so much for PortableApps.com! It was invaluable at university when studying in the library. To this day, I still use it for utilities that don't need to be installed on my desktop.
- mrsalt 6y ago
  I am also grateful for PortableApps.com. Along with Scoop and Homebrew, they make using a system without root or admin privileges a really nice experience, in both Windows and Linux. My sincere thanks to John and all the PortableApps.com contributors.
  - JohnTHaller 6y ago
    You're welcome! I'm glad it's helping you be more productive!
- JohnTHaller 6y ago
  You're welcome, I'm glad it's helped and helping you! I try to keep it growing and relevant with a more synced cloud folder/work or school laptop/keep work and personal separate bent these days.

michaelbuckbee 6y ago

Honest question: I've never really understood the back of the napkin math of how Cloudflare functions economically, which I feel would go a long way towards my understanding of why/how they were able to become such an integral and generally positive part of the Internet. Did they have some crazy in to get cheap bandwidth? Did they bet big on bandwidth prices falling? Did they figure something else out that nobody saw? Do they just to a tremendous job of migrating sites from free to paid plans?

jgrahamc 6y ago
Read our S-1, it's all in there. https://www.sec.gov/Archives/edgar/data/1477333/000119312519222176/d735023ds1.htm https://www.sec.gov/Archives/edgar/data/1477333/000119312519...
- potency 6y ago
  Off topic, but any regrets on getting involved in content policing? It always sat with me as wrong and a disturbing precedent that an internet backbone service such as yourselves would make it their business to shut down unsavory yet legal speech.
  - nix23 6y ago
    Wikileaks and Piratebay are customers of CF, so your 'legal speech' must be really something. Any links?
    - largbae 6y ago
      Maybe GP is referring to 1.1.1.2 and 1.1.1.3, the DNS resolvers that filter malware and malware+adult content respectively? Both are optional alternatives to the unfiltered 1.1.1.1 DNS service if so...
    - 236dev 6y ago
      The only ones I can think of are The Daily Stormer and 8chan

eloff 6y ago

Now that Cloudflare is also a registrar, they could pretty easily implement a nameserver failover like EasyDNS. I hope this event underscores the importance of that to them. It's worth noting Cloudflare also supports secondary DNS, but only for enterprise customers: https://blog.cloudflare.com/secondary-dns-a-faster-more-resilient-way-to-serve-your-dns-records/ https://blog.cloudflare.com/secondary-dns-a-faster-more-resi...

psim1 6y ago

What responsibility and reparative measures has Cloudflare taken for Friday's incident? Was anyone fired for the mistake?

sujinge9 6y ago
Why would firing someone make you feel like reparative measures have been taken?
- psim1 6y ago
  Reparations would show an actual sense of responsibility. Firing someone would be appropriate if they were negligent. Other measures might be more appropriate. Is it enough that any time there's a Cloudflare incident, all we get are lengthy blog posts and sorries from Cloudflare?
  - gizmo385 6y ago
    Firing people for making mistakes is just going to foster a culture of secrecy and shame. Being so quick to fire is not how you retain talent and it isn't how you foster a healthy, blameless development culture in your workplace.
  - axaxs 6y ago
    Firing people for making mistakes is a great way to watch productivity dive. The easiest way to prevent mistakes is to do nothing at all.
    - psim1 6y ago
      I understand the point being made here, but what are those affected supposed to take away? Cloudflare made a mistake that caused (x millions of dollars of lost online commerce revenues, y number of missed telehealth sessions, etc.) and since we do not punish mistakes, nothing was done. Sorry everyone!
      - 6y ago

nonbirithm 6y ago

Question: is it even possible to have DDoS protection without using a provider of it which becomes a single point of failure? Or is it maybe possible to decouple this single feature from everything else that Cloudflare provides that could take out all the sites in the future from an unrelated misconfiguration? I don't see the centralization as a positive, but I'm wondering what percentage of the websites that were taken offline see themselves as having no choice but to use Cloudflare in order to prevent themselves from being taken down anyway from malicious actors instead of by accident.

deleted 6y ago
[deleted]
divbzero 6y ago
I think you could use Cloudflare as your primary DNS provider and benefit from their DDoS protection, but also specify backup DNS name servers with a different DNS provider in case Cloudflare fails.
sschueller 6y ago
Sure, Arbour and others sell devices to deal with ddos and many isps have clusters of these which you can use. Of course this is a service that costs money.

tannhaeuser 6y ago

Whether this is an ad piece by a competitor or not, the problem with monopolies is that "the market" (if there is one) gets skewed incentives. Cloudflare has received heavy investment by FAANG (+MS) [1] before their IPO, so rather than eg Google or others with a vested interest and capability stepping up the game and invest into new IP control plane-level DDOS protection standards or similar, the situation smells more like a backdoor deal, such as an agreement to not go after a particular market segment. Let's also not forget Cloudflare in particular have been accused to host/hide the very bad boys that make protection from DDOS necessary in the first place. Whether or not that is the case, a quasi-monopoly leaves customers with no choice. [1]: https://petri.com/microsoft-google-and-others-invest-in-cloudflare https://petri.com/microsoft-google-and-others-invest-in-clou...

TedDoesntTalk 6y ago

> Turns out half the internet has a Single-Point-of-Failure called “Cloudflare” The other one is called AWS.

louwrentius 6y ago
So in the end, who cares about single point of failures?
- timbit42 6y ago
  Militaries. The internet was created so the US President could command the US military even in the case of nuclear war knocking out many of the internet's nodes.

solotronics 6y ago

Maybe it would make sense to have multiple independent sections of backbone at the BGP level. Instead of having one public AS/backbone, break it down into regions at least so that it is more confederated.

citizenpaul 6y ago

I love all the comments about fail-over for DDOS/DNS protection. What is your budget? Well we are looking at around $0.00 for our maximum allowance. Ok so single point of failure it is I guess we are done here. Companies only say they care when there is a problem, the reality is that they dont.

divbzero 6y ago
It obviously takes more than $0.00 but not that much more. It’s a matter of adding a second DNS provider and making sure you replicate DNS records manually or with AFXR. What’s really puzzling is if the same companies spend money on active/passive failover for application and database servers while overlooking DNS single point of failure.
FireBeyond 6y ago
Precisely. I remember when the CEO of my old company came to me and said (with respect to moving from a on-prem model to SaaS), "What's our SLA going to be?" "Well, what do you want it to be?" Give me a number and I'll tell you how much it will cost and how long it will take to get there.

crazygringo 6y ago

I guess this is off-topic, but the stock photo they're using is cracking me up. Guy at work... coffee cup on a tablet he's using for a coaster... except he's also drinking whiskey from a beautiful crystal glass... there's a folded paper airplane... there's just so much to unpack here, it's pretty hilarious.

hinkley 6y ago
> except he's also drinking whiskey from a beautiful crystal glass And he has left the stopper off of the decanter like some sort of animal. I think we are supposed to believe that the person here was just dicking around online, fiddling with paper, finishing his morning coffee, when all of a sudden he gets an email asking if anyone knows what's going on with the website. So he stops what he was (not) doing, puts down his coffee, and starts poking around. At which point he realizes he needs something stronger than coffee. As he is pouring his glass he is confronted with the true horror of the situation, drops the stopper on the floor and just holds his face wondering why the Universe hates him. I wonder if we can get JJ Abrams to option the movie rights.

niutech 6y ago

The solution is the Decentralized Web (DWeb), such as IPFS (https://ipfs.io https://ipfs.io), Freenet (https://freenetproject.org https://freenetproject.org), GNUnet (https://gnunet.org https://gnunet.org) or Hypercore (https://hypercore-protocol.org https://hypercore-protocol.org). We should start using them to avoid centralization and embrace freedom.

nemothekid 6y ago
Can you explain a bit more how this is a solution? Cloudflare isn't facebook. They have plenty of competitors, they don't have a massive moat, and they are almost exclusively used by businesses. Despite all of this, we should ask ourselves how we even got here. Why would companies move to DWeb, when they are already choosing to use Cloudflare instead of Fastly/Cloudfront/Akamai.
- return1 6y ago
  No salespeople to sell dweb
mattl 6y ago
How do I publish a website on one of these?
- yjftsjthsd-h 6y ago
  https://www.dgendill.com/posts/technology/2019-10-15-publish-website-to-ipfs.html https://www.dgendill.com/posts/technology/2019-10-15-publish...
cortesoft 6y ago
Can any of those handle massive scale?

surround 6y ago

Instead of trusting any DNS provider with your queries, I recommend running your own recursive resolver with Unbound. https://nlnetlabs.nl/projects/unbound/about/ https://nlnetlabs.nl/projects/unbound/about/

everfree 6y ago
But then aren't you just trusting your VPS provider or ISP instead?
- surround 6y ago
  Unfortunately, ISPs can see the domains you connect to, even if you use DNS-over-HTTPS. https://irtf.org/anrw/2019/slides-anrw19-final44.pdf https://irtf.org/anrw/2019/slides-anrw19-final44.pdf

actuator 6y ago

Does anyone know a dashboard/list for past Akamai outages. Surprisingly, I haven't seen a lot of news about Akamai downtimes. I searched on Google and the last one I found in a news report is from 2011 when its customers like Facebook, Twitter were impacted. They were a PITA to work with when I used them in the past but if they are really that good in service availability, you can have some justification for their overpriced service.

EE84M3i 6y ago
AFAIK Akamai only makes their service notifications available directly to subscribers.

louwrentius 6y ago

Who at a C-level position is going to tell anybody that the potential risk of Cloudflare (or Amazon/Azure/GCP) going down should be protected against? I would applaud them, but I wonder.

divbzero 6y ago
There’s a lot of truth to this. Cloudflare for now has achieved IBM status in that no one will be fired for choosing Cloudflare, in spite of any issues that arise.
Nasrudith 6y ago
My guess is Wall Street HFT or other financial areas with very strict unscheduled downtime penalties where they are effectively incentivised to be batshit paranoid as it would take decades for any penny pinching to remotely pay off. I don't know if many of them use Cloudflare for their domains though.

johnghanks 6y ago

lmfao this is literally a hit piece from a competitor.

jonplackett 6y ago

Similar problems are happening with crypto - yeah it’s distributed but so many people are using Coinbase that if they go down it’s going to cause a lot of problems

zelphirkalt 6y ago

Well, I usually block it anyway, because it is not only a single point of failure, but also a single point of concentration of data, that can be used to track, spy on and profile users. As such, I do not blindly trust Cloudflare. I remain sceptical, no matter how positive their public image is. Especially I would not set my DNS to cloudflare.

ampersandy 6y ago
Who do you use for DNS then that you do trust?
- yjftsjthsd-h 6y ago
  For lookups, it's not that hard to do your own recursive resolver.

mountainboy 6y ago

More importantly, CloudFlare is the world's largest man-in-the-middle. You think your TLS connection is e2e between you and the website you are visiting? Not so much... because the website has given their certs to CF. Worse, sometimes/often the connection between CF and the website is not encrypted at all.

neycoda 6y ago

When Cloudflare causes the problem they were built to solve.

deleted 6y ago

[deleted]

black_puppydog 6y ago

What I find really shocking is the abundant use of CF on piracy websites of all things. Not the serious ones of course, SciHub and library genesis are mirrored differently. But a lot of small torrent websites and such simply won't load without JS and specifically CF code. It's pretty crazy. Luckily I don't use any of those bEcAuSe IlLeGaL but still, I find it really depressing, especially when webtorrent, IPFS etc are available, and frankly many of those pages will never have to bear a load that makes CF a requirement.

jdc0589 6y ago
its not about caching or handling normal traffic. its about ddos protection. sites like that are frequent targets.
- black_puppydog 6y ago
  and there should be enough of them that that shouldn't matter.

phkahler 6y ago

If you keep much of a website simple - plain html and css - won't that reduce the need for a CDN in the first place?

deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
foxfired 6y ago
Yes, but cloudflare reach is much deeper then that. I am not a cloudflare customer but my all websites failed that day. The reason was digitalocean uses cloudflare, I use digitalocean. So apparently I depend on cloudflare.

hinkley 6y ago

High availability is an insurance game, and perhaps we need to start treating it that way. Rather than admitting that your customers need to maintain a business relationship with your competitors, you need to admit you need to maintain a business relationship with your competitors. That we need a moral equivalent of underwriting in the cloud space.

tuxninja 6y ago

AWS in 2012, DynDNS after that, now Cloudflare...I wrote about this a few years ago, the threat of the singularity of the Internet. What was distributed will be centralized again. http://tuxlabs.com/?p=430 http://tuxlabs.com/?p=430

johnklos 6y ago

I think people are Cloudflare fans simply because so many other services suck more. I gave Cloudflare a fair shake. But after hearing their lies way too many times, I'm calling them out for being deceptive and unscrupulous. After being told that sites pretending to host Adobe Flash updaters and pretending to be Bank of America can't be taken down by Cloudflare because of their rights to free speech, I knew their attempts to pretend to be one of us, attempts to pretend to care, were nothing but bullshit. They claim they don't host. If hosting DNS is not hosting, then what do you call DNS hosting? You literally CANNOT use their abuse web form to report domains which use Cloudflare just for hosting DNS. They do not handle abuse sent to their abuse email address (they simply send a form response saying to spend ten minutes filling out their crappy form that has all sorts of problems). Of course, their web proxy services are also "not hosting", even though they're protecting all sorts of scammers. So why should we think they're not bullshitting us when they run 1.1.1.1 and tell us they're not logging? Why should we trust them more than our ISPs by running DoH through them? They WANT us to be dependent on them, because the more control they have, the more money they can make. It's dangerous, and they've shown they have no honor. I've genuinely tried to correspond with them on Twitter, and they excel at not answering the question asked but instead just diverting. It's scummy, unprofessional behavior and I encourage everyone to consider whether they deserve anyone's data or business.

james412 6y ago
> They WANT us to be dependent on them I think this is an important point about CloudFlare that can't be made often enough. It's been some years since I first noticed, and it seems as true today as it was then: wedging themselves into core Internet services and data flows seems to be an intentional part of CloudFlare's strategy. There is no case given the architecture of the Internet where one company need be exposed to so many traffic flows from millions of people. The search engines got there first and we eventually stopped complaining, but this does not make it any more justifiable to copy the model. What reason would a company have for desiring this outcome? We know Google can detect and predict flu outbreaks. Imagine what is possible when you have every click on every target web site. There is a fair chance their data is already approaching the comprehensiveness of Google, and I'd be surprised, if not disappointed to learn they were not already working on unannounced (now or eternally) internal intelligence products based on that data. There are simply too many pockets who would be willing to pay for it.

ivanvanderbyl 6y ago

Pretty poor form calling out a competitor for something like this. Not the first time a DNS provider has done this, and won't be the last.

knorker 6y ago

The other half is on AWS?

miki123211 6y ago

Cloudflare is horrible for blind people. Screen readers, the programs that use synthesized speech to tell us what's on the screen, cannot read images. Good captchas usually have audio equivalents (which come with their own set of problems), but this one doesn't. If you're blind and flagged by Cloudflare for some reason, you're cut off from accessing half the internet, potentially critical banking/governmental/medical/communications/educational services. We rely on the internet way more than our sighted peers, so this is very important. This has recently happened to me on a few sites, fortunately not critical ones, but it was not a pleasant experience nonetheless. CF engineers, please fix this ASAP. I'm surprised there still isn't a huge lawsuit over this, as this is clearly violating all sorts of laws.

deleted 6y ago
[deleted]
strombofulous 6y ago
I use audio captchas. Google will usually only let you do 2 or 3 before banning you and making you do image-based ones. I'm pretty sure the button is just there to make it seem accessible.
- dependenttypes 6y ago
  I had exactly the same issue. I am surprised how neither cloudflare nor google have been sued for making most of the web inaccessible to people with disabilities.
  - lopmotr 6y ago
    Shouldn't that liability be with the operators of websites that use those captchas if they're required to be blind-accessible? If they get sued, it'll apply pressure up the supply chain.
    - kbenson 6y ago
      I agree. You don't sue the contractor because they built to the specification you provided. At the same time, if Google is advertising Recaptcha as accessible and it's not really, then they need to be held accountable for that, because that has a real impact on huge swaths of the internet, and especially for sites that try to do the right thing and find out Google has screwed them.

valuearb 6y ago

“Cloudflare apparently fat-fingered a routing update and sent all of their global traffic to a single POP, vaporizing it almost instantly.” Made me chuckle, as it gave me the image of a large server in some massive server farm glowing red, then bursting in a massive burst of light as dozens of bearded Sysadmins run out of the building screaming.

sciurus 6y ago
In my experience the sysadmins would be running in the opposite direction.
- Baeocystin 6y ago
  As usual, a relevant one: https://xkcd.com/705/ https://xkcd.com/705/

Uhhrrr 6y ago

> But if you want to use a preferred DNS provider, such as Cloudflare, who use their DNS responses to optimize your website proxy. That works best most of the time, so then you want to go with an active/passive model that will step back when things are going according to plan, and then when these periodic network cataclysms do occur (and they will), they step into the breach and update your nameservers so that you at least stay up until the crisis is over. Copy editors are cheap and your reputation shouldn't be.

WarOnPrivacy 6y ago

This is the 3rd DNS related outage I've seen in a week. Frountier Communications lost their DNS. Trying to recall what the 3rd one was. My Unbound resolver round-robins DNS-over-TLS requests, between Cloudflare & Quad9. Cloudflare's outage never impacted us that I could tell. Nevertheless, I am reminded that I ought to add a couple of DoT providers (who aren't Google). Not sure who else came online since I setup.

anthk 6y ago

Cloudflare doesn't work without JS, even for static jsless sites. It's a pest.

aabbcc1241 6y ago

There are some (research) work on proof-of-human protocol, interesting direction

aww_dang 6y ago

Cloudflare is hell when you need to load scripts and css 403'd by them. Here's my work around: 1) open developer tools 2) refresh page 3) move tab into a new window 4) find a blocked resource in the network tab of developer tools and open it in a new tab 5) verify humanity 6) repeat step 4 and sometimes 5 for each resource the page requires 7) move the problematic site out of the new window and close all of these tabs at once It is a terrible experience. CF devs and publishers have no idea how inconvenient their service is. I wonder if they have ever lived in a region where every ISP is both incompetent and listed in the CBL? I frequently skip sites that use CF. The captchas are obnoxious. The entire concept of a webapp firewall seems a bit backwards to me. Developers should fix their insecure applications.

tonyfader 6y ago

you can warn people, but they don't listen...

erichocean 6y ago

"Turns out over half the Internet has a single point of failure called BGP."

imvetri 6y ago

Haha

dschuetz 6y ago

When I was figuring out how DNS works back 10 years ago, I was told "Why don't you just use 8.8.8.8 everywhere? Why do you need your own DNS server for anyway?" every single time when I asked a very specific configuration question. Some years later 8.8.8.8 was just replaced with 1.1.1.1 with same critical counter questions instead of helping. So, it appears to me, instead of understanding DNS and routing, people and tech-bro businesses take shortcuts by using centralized infrastructure for their systems, creating dangerous configurations. Now, Cloudflare have made a mistake, and half the Internet crumbled down. See the irony?

satokema_work 6y ago

CHEAP V1AGRA N0W!!11

rchaud 6y ago

Starting yesterday, I have been getting Cloudflare's hCaptcha image identification quizzes on every site that uses them. Googling the issue indicates that the IP address range may have been blacklisted, if say my laptop (a Mac that doesn't visit any shady sites) or my router has been compromised and is now running a botnet or something. It seems unlikely because my phone can access sites on wifi fine, and again my Mac doesn't appear to have any malware. Could it be that the outage is somehow relate to Cloudflare putting these captchas up as a precautionary measure?