7 ms·

It really defies the original vision of the internet to have so many services depend on a single company. Almost every news site I was reading dropped off at on

It really defies the original vision of the internet to have so many services depend on a single company. Almost every news site I was reading dropped off at once. I thought for a second that I lost internet in my own house.

rickyc091 6y ago
Same here. Rebooted the router and modem thinking it was me, but my phone was still on wifi then realized it was probably my cloudflare DNS.
- asadlionpk 6y ago
  This! I got all sorts of alerts from pingdom and my laptop refused to get online. Pure Panic!
- spiritplumber 6y ago
  Pihole is your friend.
  - cls59 6y ago
    Yeah, Pihole made it super easy to cut over to Quad-9 once I figured out what the problem was.
  - rickyc091 6y ago
    Looks like I got another weekend project.
  - newhotelowner 6y ago
    I have a pihole. It didn't help.
- xen2xen1 6y ago
  Yup, reinforces the thought that you never have both DNS servers with the same service.
jeremyjh 6y ago
Yes its really odd that core backbone providers can go down and everything works like its supposed to. Even trans-pacific cables can be cut and things will usually work with only increased latency. But there is not much redundancy for many companies at this layer; having redundant DNS providers is I'm sure possible but not something we think about very often, and of course many of the sites that are down are depending on the proxy and DOS mitigation services. On my home network I use Google as a backup DNS provider so the whole internet didn't go dark for me, but I don't have a backup DNS host for my company's DNS records.
- kiobu 6y ago
  I imagine most people would never expect something like this to happen, so having a fallback option when Cloudflare has a huge interruption of service like this is just unthinkable.
  - macNchz 6y ago
    All the major cloud infrastructure providers have had outages of varying severity at one point or another...it's something you'd want to take into account for, say, a system that remote controls life-critical devices, but likely isn't worth the engineering time and added complexity for a productivity or social app with a small userbase. Working on many of the latter over the years I've generally said "well if {major cloud provider} is down, the internet is going to be all messed up for a bit anyway, so we'll accept the risk of being down when they're down, and reassess whether that keeps making sense as we grow."
    - dijit 6y ago
      >"well if {major cloud provider} is down, the internet is going to be all messed up for a bit anyway, so we'll accept the risk of being down when they're down, and reassess whether that keeps making sense as we grow." This is a very common pattern and falls into the 'nobody got fired for buying cisco/microsoft/intel' trap. I have two issues with it; 1) It entrenches the largest provider. You would not extend the same leniency's of outages to the third best cloud provider, this means that people will just keep pushing the monopoly forward. Even if the uptime or service is actually better on another provider. 2) You create a tight coupling of monocultures; Simply put: You slowly erode the internet. Your site becomes an application in a distributed mainframe operated by a tiny minority of tech companies universally based in the US. Why is this a problem? I could give moral answers here but I thing pragmatic ones are more convincing.. Giving ownership of the internet to the few gives them the ability to set the rules. If you're on Amazon's AWS, what's to say they don't inspect your e-commerce systems and incorporate your business logic into their amazon.com shopping experience. They do this to their marketplace and create competing products already[0]. If you're doing really well, why not just drop a few packets here and there? I mean, they wont.. you're paying, right? Hell, if you do super well they can just change the rules and make it so your services get expensive in the exact way you use them, or even legislate you off the platform entirely. Probably wont happen, but it's a lot of trust you have to admit, and people shit on Apple for having that kind of power, and Apple is not even competing in the same market as most people on this site. If you're on google cloud (which, I'm a fan of btw) and you feed an ML model, well, you paid for it but why shouldn't they also have a copy.. after all, it's green to do so! Their bigdata platform? Google loves data! Feed the beast. [0]: https://fortune.com/2016/04/20/amazon-copies-merchants/ https://fortune.com/2016/04/20/amazon-copies-merchants/
remmargorp64 6y ago
I consider DNS and the way how top level domains are handled to be one of the weakest parts of our current Internet design. We REALLY need a truly decentralized, distributed DNS system that is not owned by private entities.
- hpfr 6y ago
  https://handshake.org https://handshake.org is pretty interesting.
  - spenczar5 6y ago
    The "decentralized internet" folks always talk a lot about fighting corporate control. I think they should spend more time talking about resiliency and blast-radius reduction.
- ghastmaster 6y ago
  I just recently ran across this. I wonder how much performance would be degraded. https://ieeexplore.ieee.org/document/7530014/authors#authors https://ieeexplore.ieee.org/document/7530014/authors#authors > Unlike previous DNS replacement proposals, D 3 NS is reverse compatible with DNS and allows for incremental implementation within the current system.
- the8472 6y ago
  DNS is far less of a single point of failure and more decentralized than cloudflare. Nameservers can and are operated redundantly via simple, resolver-side round-robin scheduling and the TLD servers should have longer TTLs that allow plenty of caching. The rootzone even has anycast thanks to using UDP. Take a moment to look at DoH and laugh. You can also also register your domain on multiple TLDs.
- xen2xen1 6y ago
  DNS is decentralized, it's just not when everyone goes with one big service.
  - 6y ago
Algent 6y ago
And the worst is if you try to raise concerns about cloudflare now it get brushed of as "cf already proxy half the internet, if it goes down our stuff will be minor concern".
- Can_Not 6y ago
  That's true, but what's a free or low cost alternative for DDoS protection for a small webapp?
lumberingjack 6y ago
Same here. I'm working at an auto parts store looking though ASE parts sites and it was like well close up the store the catalogs are missing RN.
hn_throwaway_99 6y ago
Agreed, but I think people really underestimated the forces at work that would cause so much consolidation into a couple internet giants. The original idea was that with the barrier to entry being so low, anyone and everyone could set up their own websites, mail servers, etc. But with it being so easy to compare and contrast service (i.e. the market being so open), it means that the competitive forces naturally consolidate to a winner-take-all model. If when starting out Cloudflare was just 5% better than the competition, it could have easily taken the vast majority of the mindshare on the internet. Couple that with the fact that there are huge advantages with scale to a business like Cloudflare's, and it's not hard to see how so much of the internet has become dependent on it.
Meekro 6y ago
Agreed, but the real problem is DDoS and nobody seems to know how to globally solve it. Fighting DDoS is expensive, so you see consolidation. It's well and good to live in a tiny farming town but when raiders start attacking every week, those castle walls and guards start to look really appealing.
- labawi 6y ago
  That's what we get for externalizing costs. It's not hard to track down sources, but network operators usually let it be, hence the incentives are probably counter-productive.
- schoolornot 6y ago
  It's nice that Cloudflare provides their services for free but scrubbing has existed for a long time. With your own address space and an appropriate budget it's not difficult to have Cloudflare/Akamai/AWS announce your IP space with a higher weight than a direct path to your infrastructure. That will give you a little bit more fault tolerance for incidents like these.
cortesoft 6y ago
I don't understand why the big companies don't always have at least two CDN providers, so they can failover to another one if something like this happens. I know a lot of big companies do, but I am always surprised when you see ones that don't.
- LoSboccacc 6y ago
  the DNS itself is not as easy to duplicate across multiple provider, with CF DNS down having a backup CDN wouldn't have helped
  - cortesoft 6y ago
    This isn't true... you can certainly do redundant dns with automatic failover between providers. Just set up NS records pointing to different providers.
    - lima 6y ago
      That's not easy, you need to set up replication.
      - cortesoft 6y ago
        Seems worth doing
thathndude 6y ago
My CRM was nonfunctional. That’s some critical infrastructure for me. And then I’m wondering, is it me or is it my CRM. Turns out it’s door #3 - cloudflare

rickyc091 6y ago

Same here. Rebooted the router and modem thinking it was me, but my phone was still on wifi then realized it was probably my cloudflare DNS.

asadlionpk 6y ago
This! I got all sorts of alerts from pingdom and my laptop refused to get online. Pure Panic!
spiritplumber 6y ago
Pihole is your friend.
- cls59 6y ago
  Yeah, Pihole made it super easy to cut over to Quad-9 once I figured out what the problem was.
- rickyc091 6y ago
  Looks like I got another weekend project.
- newhotelowner 6y ago
  I have a pihole. It didn't help.
xen2xen1 6y ago
Yup, reinforces the thought that you never have both DNS servers with the same service.

jeremyjh 6y ago

Yes its really odd that core backbone providers can go down and everything works like its supposed to. Even trans-pacific cables can be cut and things will usually work with only increased latency. But there is not much redundancy for many companies at this layer; having redundant DNS providers is I'm sure possible but not something we think about very often, and of course many of the sites that are down are depending on the proxy and DOS mitigation services. On my home network I use Google as a backup DNS provider so the whole internet didn't go dark for me, but I don't have a backup DNS host for my company's DNS records.

kiobu 6y ago
I imagine most people would never expect something like this to happen, so having a fallback option when Cloudflare has a huge interruption of service like this is just unthinkable.
- macNchz 6y ago
  All the major cloud infrastructure providers have had outages of varying severity at one point or another...it's something you'd want to take into account for, say, a system that remote controls life-critical devices, but likely isn't worth the engineering time and added complexity for a productivity or social app with a small userbase. Working on many of the latter over the years I've generally said "well if {major cloud provider} is down, the internet is going to be all messed up for a bit anyway, so we'll accept the risk of being down when they're down, and reassess whether that keeps making sense as we grow."
  - dijit 6y ago
    >"well if {major cloud provider} is down, the internet is going to be all messed up for a bit anyway, so we'll accept the risk of being down when they're down, and reassess whether that keeps making sense as we grow." This is a very common pattern and falls into the 'nobody got fired for buying cisco/microsoft/intel' trap. I have two issues with it; 1) It entrenches the largest provider. You would not extend the same leniency's of outages to the third best cloud provider, this means that people will just keep pushing the monopoly forward. Even if the uptime or service is actually better on another provider. 2) You create a tight coupling of monocultures; Simply put: You slowly erode the internet. Your site becomes an application in a distributed mainframe operated by a tiny minority of tech companies universally based in the US. Why is this a problem? I could give moral answers here but I thing pragmatic ones are more convincing.. Giving ownership of the internet to the few gives them the ability to set the rules. If you're on Amazon's AWS, what's to say they don't inspect your e-commerce systems and incorporate your business logic into their amazon.com shopping experience. They do this to their marketplace and create competing products already[0]. If you're doing really well, why not just drop a few packets here and there? I mean, they wont.. you're paying, right? Hell, if you do super well they can just change the rules and make it so your services get expensive in the exact way you use them, or even legislate you off the platform entirely. Probably wont happen, but it's a lot of trust you have to admit, and people shit on Apple for having that kind of power, and Apple is not even competing in the same market as most people on this site. If you're on google cloud (which, I'm a fan of btw) and you feed an ML model, well, you paid for it but why shouldn't they also have a copy.. after all, it's green to do so! Their bigdata platform? Google loves data! Feed the beast. [0]: https://fortune.com/2016/04/20/amazon-copies-merchants/ https://fortune.com/2016/04/20/amazon-copies-merchants/

remmargorp64 6y ago

I consider DNS and the way how top level domains are handled to be one of the weakest parts of our current Internet design. We REALLY need a truly decentralized, distributed DNS system that is not owned by private entities.

hpfr 6y ago
https://handshake.org https://handshake.org is pretty interesting.
- spenczar5 6y ago
  The "decentralized internet" folks always talk a lot about fighting corporate control. I think they should spend more time talking about resiliency and blast-radius reduction.
ghastmaster 6y ago
I just recently ran across this. I wonder how much performance would be degraded. https://ieeexplore.ieee.org/document/7530014/authors#authors https://ieeexplore.ieee.org/document/7530014/authors#authors > Unlike previous DNS replacement proposals, D 3 NS is reverse compatible with DNS and allows for incremental implementation within the current system.
the8472 6y ago
DNS is far less of a single point of failure and more decentralized than cloudflare. Nameservers can and are operated redundantly via simple, resolver-side round-robin scheduling and the TLD servers should have longer TTLs that allow plenty of caching. The rootzone even has anycast thanks to using UDP. Take a moment to look at DoH and laugh. You can also also register your domain on multiple TLDs.
xen2xen1 6y ago
DNS is decentralized, it's just not when everyone goes with one big service.
- 6y ago

Algent 6y ago

And the worst is if you try to raise concerns about cloudflare now it get brushed of as "cf already proxy half the internet, if it goes down our stuff will be minor concern".

Can_Not 6y ago
That's true, but what's a free or low cost alternative for DDoS protection for a small webapp?

lumberingjack 6y ago

Same here. I'm working at an auto parts store looking though ASE parts sites and it was like well close up the store the catalogs are missing RN.

hn_throwaway_99 6y ago

Agreed, but I think people really underestimated the forces at work that would cause so much consolidation into a couple internet giants. The original idea was that with the barrier to entry being so low, anyone and everyone could set up their own websites, mail servers, etc. But with it being so easy to compare and contrast service (i.e. the market being so open), it means that the competitive forces naturally consolidate to a winner-take-all model. If when starting out Cloudflare was just 5% better than the competition, it could have easily taken the vast majority of the mindshare on the internet. Couple that with the fact that there are huge advantages with scale to a business like Cloudflare's, and it's not hard to see how so much of the internet has become dependent on it.

Meekro 6y ago

Agreed, but the real problem is DDoS and nobody seems to know how to globally solve it. Fighting DDoS is expensive, so you see consolidation. It's well and good to live in a tiny farming town but when raiders start attacking every week, those castle walls and guards start to look really appealing.

labawi 6y ago
That's what we get for externalizing costs. It's not hard to track down sources, but network operators usually let it be, hence the incentives are probably counter-productive.
schoolornot 6y ago
It's nice that Cloudflare provides their services for free but scrubbing has existed for a long time. With your own address space and an appropriate budget it's not difficult to have Cloudflare/Akamai/AWS announce your IP space with a higher weight than a direct path to your infrastructure. That will give you a little bit more fault tolerance for incidents like these.

cortesoft 6y ago

I don't understand why the big companies don't always have at least two CDN providers, so they can failover to another one if something like this happens. I know a lot of big companies do, but I am always surprised when you see ones that don't.

LoSboccacc 6y ago
the DNS itself is not as easy to duplicate across multiple provider, with CF DNS down having a backup CDN wouldn't have helped
- cortesoft 6y ago
  This isn't true... you can certainly do redundant dns with automatic failover between providers. Just set up NS records pointing to different providers.
  - lima 6y ago
    That's not easy, you need to set up replication.
    - cortesoft 6y ago
      Seems worth doing

thathndude 6y ago

My CRM was nonfunctional. That’s some critical infrastructure for me. And then I’m wondering, is it me or is it my CRM. Turns out it’s door #3 - cloudflare