4 ms·

If you’re waiting for the inevitable “this is why we shouldn't centralize everything”; this is that comment. As much as I like cloudflare and easy to use stuff

by dijit 3y ago

If you’re waiting for the inevitable “this is why we shouldn't centralize everything”; this is that comment. As much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope. luckily this time its just administrative interfaces and supplemental services that assist in development and that hopefully aren't used in production. People will defend cloudflare here with the old saying “Operating at scale is hard” but simultaneously holding the opinion that operating services themselves is equally hard. I guess the argument is that outsourcing is easier, but then you are very much out of control. “the internet” in its practical entirety going down becomes more possible every passing day.

ta1243 3y ago
Operating at scale is hard. However the internet doesn't require people to "operate at scale". That's the beauty. It's a mash of different (relatively) small autonomous systems, with many hosts. If any host goes down, or any network goes down, the system ignores it, or reroutes it.
- baq 3y ago
  In practice, cloudflare or AWS down means half the internet doesn't work at all. The half that works - routing around problematic networks - isn't very helpful when the problematic networks are the useful ones.
  - ta1243 3y ago
    Depends what you want to get to. HN doesn't use any centralised services. If someone chooses to host their service relying on a single unreliable point of failure that's on them. Plenty more hosts on the web.
    - djbusby 3y ago
      Isn't HN centralized on like two servers in CA?
    - solardev 3y ago
      > Depends what you want to get to. HN doesn't use any centralised services. It's also one of the least reliable and performant public forums I've ever used. It goes down or slows down several times a week for me. Wish they'd just move it to the cloud instead of occasionally upgrading the bare metal...
      - inemesitaffia 3y ago
        They used AWS when they had lost their SSD's
    - shanemhansen 3y ago
      Well technically they do, but honestly the number of companies that need to worry about redundant DNS is probably .001%. It does matter for the some people, but as someone from SRE once said "100% availability is the wrong target for basically everything". dig SOA news.ycombinator.com ; <<>> DiG 9.18.16-1-Debian <<>> SOA news.ycombinator.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10818 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;news.ycombinator.com. IN SOA ;; AUTHORITY SECTION: ycombinator.com. 900 IN SOA ns-225.awsdns-28.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 39 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Wed Oct 25 10:52
- mschuster91 3y ago
  > However the internet doesn't require people to "operate at scale". The internet itself does not, yes, but the internet as a system, filled to the brim with bad actors ranging from 14 year old skiddies with Metasploit over hardened Russian ransomware groups to virtually all nations' secret services and militaries does. As soon as you start a server with a globally routed IP address, it will get hammered from all kinds of scanners. Anything on port 80/443 will get bombarded with Wordpress, Drupal and log4j attempts, anything on port 22 will get hounded by people attempting credential stuffing, and anything on port 25/587 by spammers and anyone trying to exploit Exchange/Exim vulnerabilities. It's a nightmare to keep up with that crap alone, and you're one forgotten update away from getting your server 0wned by someone using Shodan and whatnot. And then, once your site, whatever you operate - I've seen as an admin DDoS extortion attacks happen against small sites for dog walker services, pizza parlors or random bloggers as well as large car dealerships - finds itself some opponent for whatever reason, it gets nasty. Be it for political reasons (people don't like what the blogger writes), for the lulz (people don't like a Twitch streamer so they DDoS their sponsors), or for financial extortion, doesn't matter in a time where you can order such attacks for less than the price of your average McDonalds visit. There is no, utterly no, alternative to the giants (Cloudflare, AWS CloudFront, Akamai). It's sad that it has come to that point, but unless we get the most obnoxious sources of such attacks - enemy nation states (China, Russia) and compromised domestic IoT devices/servers - off of the Internet, there is really no viable alternative.
  - ta1243 3y ago
    > There is no, utterly no, alternative to the giants (Cloudflare, AWS CloudFront, Akamai). Yet my servers are fine without such services.
shepherdjerred 3y ago
What's the impact for most sites if they go down a few times a year due to AWS/CloudFlare?
- dijit 3y ago
  Whats the impact if the entire internet is down for a couple of hours? What if its more than a few hours because theres a natural monopoly and a dearth of capable people to run systems independently - thus no strong external incentive for rapid response outside of punitive SLAs? How much soft power do you think such an entity would have on the world stage.
  - solardev 3y ago
    > Whats the impact if the entire internet is down for a couple of hours? People get some off-screen time? Life goes on. > What if its more than a few hours because theres a natural monopoly?[...snip...] How much soft power do you think such an entity would have on the world stage. They're just commodity infra providers. The Facebook and Twitters of the world have way more power and influence. This isn't even a hypothetical. The big cloud providers have been around for what, a decade or so now? How many times has the world collapsed with every Cloudflare or AWS downtime? It's just not as big a deal as you paint it to be. These providers going down is like the digital equivalent of a highway crash -- a small inconvenience for a few hours, then forgotten about in a day or two.
  - shepherdjerred 3y ago
    > Whats the impact if the entire internet is down for a couple of hours? Okay, so obviously a lot of "important" stuff happens via the Internet. You could come up with some financial impact if the Internet went down for some time. Harder to measure: A lot of people would be inconvenienced or suffer. Some number of people would die indirectly, e.g. due to emergencies/lack of communication. People are inconvenienced and suffer and die all of the time. It's part of life. I'm not advocating for more death and suffering, but I am saying that, as another comment said, "life goes on". --- Put another way, if the Internet disappeared for a week, month, year, what would happen? It'd suck and we'd adjust, but we (humanity as a whole) would be fine.
gjsman-1000 3y ago
Most people blame Cloudflare and Amazon for centralizing everything. I actually disagree; the centralization is a symptom of severe, systemic, unfixable flaws in the design of the internet itself. At least we didn't let AOL define internet.
- djbusby 3y ago
  Such as?
  - gjsman-1000 3y ago
    1. DDoS Attacks 2. Port scanning attacks 3. Geolocation attacks (you shouldn't run a server from your home, when your IP Address could be used to find you; be it from a hack or an abuse of the legal process) 4. The US internet relying on a backbone mostly operated by 4 companies (meaning a direct connection is a massive perk) For as much as we moan about Amazon and Cloudflare centralizing things; believe me, just taking out Level 3 / Lumen would be almost an act of war and would affect every ISP and cloud provider. If you took out Lumen, Verizon Enterprise Solutions, and AT&T, just those 3 companies; that's 2.35 million kilometers of fiber going dark. For reference, Comcast has "only" 150,000 km.
    - djbusby 3y ago
      Those are problem and symptoms - not design flaws. How could one design a system that doesn't let DDoS exist on the network?
    - silisili 3y ago
      I actually have a lot of respect for the big backbones. If Goog/MS/Amz owned them all, they'd find some way to wall it off from everyone else. If Comcast or Charter owned them all, our internet would cost about $1000 a month by now. All things considered, IMO they've been fairly good and fair stewards.
- justin_oaks 3y ago
  I'd appreciate if you'd elaborate on that. (Edit: I see you elaborated somewhat in another comment posted before I finished this comment) I tend to think centralization is more about people's tendencies than it is internet design. As I see it, centralization on the internet happens as the emergent behavior of all of the elements in the system and the forces that exist. In other words, it's the natural result of incentives. People use centralized services because - prices (economies of scale) - network effects ("I can't certain people without WhatsApp") - social pressures ("People look at me weird when I say I don't have an Instagram") - visibility ("There are other CRMs than SalesForce?") - making choices easy ("I don't want to choose between all these options. I'll pick the most familiar") - safety ("Nobody got fired for choosing IBM") - convenience ("It's a pain to self-host") - security ("Google probably protects data better than I can") - etc
deleted 3y ago
[deleted]
SheinhardtWigCo 3y ago
The current situation is preferable to the before times, where any website could be nuked from orbit by anyone with a mild grievance. Cloudflare has succeeded in effectively shutting down that attack vector, and that’s worth the risk of occasional brief but broad outages, at least to me.
- dijit 3y ago
  There were mitigations for this but it was usually on “IT”s budget and IT was considered a cost center. Really the best innovation in the last 10 years has been the penchant for putting things on development budgets instead which loosened the purse strings considerably. I wish I had sufficient data to back this up but for context; I was part of a small infrastructure team running a little over 1% of all web traffic at one point in a pair of colos (with DDoS protection) for less than £5MM on infra costs. Now I can easily be in the 6 digits with just a CRUD app
rafaelmn 3y ago
If you think internet infrastructure is centralized, boy are you going to be depressed about AI powered services. Those are much more volatile (compute heavy and not enough compute to meet peaks), harder to setup/maintain and benefit greatly from sharing infrastructure for hardware utilization. Everything is moving in that direction.
pvg 3y ago
the inevitable It's readily evitable, just choose not to write the generic, repetitive thing you already know is generic and repetitive.
solardev 3y ago
Is it really that bad? Services go down for a few minutes, maybe a few hours? A thousand other companies are down at the same time. Grab lunch, come back, and it'll have fixed itself. Nobody blames you because it's just another provider outage -- whether it's Cloudflare or AWS or Microsoft or Gmail doesn't really matter, it's a convenient outsourcing of not just network admin but also blame. The alternative is, what, doing it all in-house with a bunch of people on pager duty? Few companies can afford that. Diversifying across multiple cloud providers? Sure, if multi-nines uptime is really critical for your business. For most small-med businesses, it's more valuable to be able to rapidly build on top of existing, professionally-maintained infrastructure in order to deliver some unique business value on top of it, than to recreate that basic network infra themselves. If you're not a FAANG, going down for a few hours a year isn't really a big deal. Especially when you can just blame it on an overall "internet outage at some big provider" rather than some internal fuck-up.
- csomar 3y ago
  I don't think the problem is a few minutes of down time but all services going down at the same time. You can't order food (Uber down), your email down, your TV down, maybe even the traffic control?
  - solardev 3y ago
    Ok, and...? (Edit: Not to be snarky, but really, so what if that happens? Same thing happens if there's a power outage in your neighborhood. Is it a big deal?)
    - king_phil 3y ago
      Are power outages even a thing? My PC at home (germany) is always on, so I'd notice, and I'm involved with energy at our data center (Frankfurt, Germany) and home and external data center had no outage for the last 20 years, at a minimum.
      - solardev 3y ago
cj 3y ago
I think the solution here isn’t “don’t use Cloudflare” … IMO if you need high availability and can’t tolerate downtime, then deploy and split load between 2 or more providers (e.g. Cloudflare, Fastly) and 2 or more clouds. The centralization around AWS and GCP and Azure is more concerning to me than the centralization around Cloudflare. It’s not obvious to me why lots of people using Cloudflare is any worse than lots of people using AWS. If we go even deeper, the entire internet runs on a limited number of undersea cables connecting continents. That in particular has always seemed like the most vulnerable aspect of the internet, considering a targeted attack on < 100 locations could meaningfully cut off internet access for millions (billions?) of people for an extended period of time.
- jabart 3y ago
  This. If downtime is that big of an issue, you should be on Cloudflare's business/enterprise plan which allows CNAME DNS, have redundant DNS providers with 30s TTLs that flip over to another provider. Better yet if you can 50/50 the traffic and 100/0 it when something goes bad. Downside is cost and DNSSEC has an issue with multiple keys. I think there was an RFC on that though I'm not sure it's well supported yet.
- pixl97 3y ago
  >then deploy and split load between 2 or more providers (e.g. Cloudflare, Fastly) How does this work when most edge providers manage your DNS to point at the available servers they have? Also, if you "HAVE" to be online, then you're the type of target that's at big risk of DDOS which complicates this even further.
  - wbl 3y ago
    Usually each one will give you a CBAME, your DNS or Cedexic decides, sometimes per client. It's a pretty common setup although can complicate config changes and taking advantage of new features.
swader999 3y ago
It's nice to at least be considering blast radius in a figurative manner, hope that lasts.

by dijit 3y ago

ta1243 3y ago

Operating at scale is hard. However the internet doesn't require people to "operate at scale". That's the beauty. It's a mash of different (relatively) small autonomous systems, with many hosts. If any host goes down, or any network goes down, the system ignores it, or reroutes it.

baq 3y ago
In practice, cloudflare or AWS down means half the internet doesn't work at all. The half that works - routing around problematic networks - isn't very helpful when the problematic networks are the useful ones.
- ta1243 3y ago
  Depends what you want to get to. HN doesn't use any centralised services. If someone chooses to host their service relying on a single unreliable point of failure that's on them. Plenty more hosts on the web.
  - djbusby 3y ago
    Isn't HN centralized on like two servers in CA?
  - solardev 3y ago
    > Depends what you want to get to. HN doesn't use any centralised services. It's also one of the least reliable and performant public forums I've ever used. It goes down or slows down several times a week for me. Wish they'd just move it to the cloud instead of occasionally upgrading the bare metal...
    - inemesitaffia 3y ago
      They used AWS when they had lost their SSD's
  - shanemhansen 3y ago
    Well technically they do, but honestly the number of companies that need to worry about redundant DNS is probably .001%. It does matter for the some people, but as someone from SRE once said "100% availability is the wrong target for basically everything". dig SOA news.ycombinator.com ; <<>> DiG 9.18.16-1-Debian <<>> SOA news.ycombinator.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10818 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;news.ycombinator.com. IN SOA ;; AUTHORITY SECTION: ycombinator.com. 900 IN SOA ns-225.awsdns-28.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 39 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Wed Oct 25 10:52
mschuster91 3y ago
> However the internet doesn't require people to "operate at scale". The internet itself does not, yes, but the internet as a system, filled to the brim with bad actors ranging from 14 year old skiddies with Metasploit over hardened Russian ransomware groups to virtually all nations' secret services and militaries does. As soon as you start a server with a globally routed IP address, it will get hammered from all kinds of scanners. Anything on port 80/443 will get bombarded with Wordpress, Drupal and log4j attempts, anything on port 22 will get hounded by people attempting credential stuffing, and anything on port 25/587 by spammers and anyone trying to exploit Exchange/Exim vulnerabilities. It's a nightmare to keep up with that crap alone, and you're one forgotten update away from getting your server 0wned by someone using Shodan and whatnot. And then, once your site, whatever you operate - I've seen as an admin DDoS extortion attacks happen against small sites for dog walker services, pizza parlors or random bloggers as well as large car dealerships - finds itself some opponent for whatever reason, it gets nasty. Be it for political reasons (people don't like what the blogger writes), for the lulz (people don't like a Twitch streamer so they DDoS their sponsors), or for financial extortion, doesn't matter in a time where you can order such attacks for less than the price of your average McDonalds visit. There is no, utterly no, alternative to the giants (Cloudflare, AWS CloudFront, Akamai). It's sad that it has come to that point, but unless we get the most obnoxious sources of such attacks - enemy nation states (China, Russia) and compromised domestic IoT devices/servers - off of the Internet, there is really no viable alternative.
- ta1243 3y ago
  > There is no, utterly no, alternative to the giants (Cloudflare, AWS CloudFront, Akamai). Yet my servers are fine without such services.

shepherdjerred 3y ago

What's the impact for most sites if they go down a few times a year due to AWS/CloudFlare?

dijit 3y ago
Whats the impact if the entire internet is down for a couple of hours? What if its more than a few hours because theres a natural monopoly and a dearth of capable people to run systems independently - thus no strong external incentive for rapid response outside of punitive SLAs? How much soft power do you think such an entity would have on the world stage.
- solardev 3y ago
  > Whats the impact if the entire internet is down for a couple of hours? People get some off-screen time? Life goes on. > What if its more than a few hours because theres a natural monopoly?[...snip...] How much soft power do you think such an entity would have on the world stage. They're just commodity infra providers. The Facebook and Twitters of the world have way more power and influence. This isn't even a hypothetical. The big cloud providers have been around for what, a decade or so now? How many times has the world collapsed with every Cloudflare or AWS downtime? It's just not as big a deal as you paint it to be. These providers going down is like the digital equivalent of a highway crash -- a small inconvenience for a few hours, then forgotten about in a day or two.
- shepherdjerred 3y ago
  > Whats the impact if the entire internet is down for a couple of hours? Okay, so obviously a lot of "important" stuff happens via the Internet. You could come up with some financial impact if the Internet went down for some time. Harder to measure: A lot of people would be inconvenienced or suffer. Some number of people would die indirectly, e.g. due to emergencies/lack of communication. People are inconvenienced and suffer and die all of the time. It's part of life. I'm not advocating for more death and suffering, but I am saying that, as another comment said, "life goes on". --- Put another way, if the Internet disappeared for a week, month, year, what would happen? It'd suck and we'd adjust, but we (humanity as a whole) would be fine.

gjsman-1000 3y ago

Most people blame Cloudflare and Amazon for centralizing everything. I actually disagree; the centralization is a symptom of severe, systemic, unfixable flaws in the design of the internet itself. At least we didn't let AOL define internet.

djbusby 3y ago
Such as?
- gjsman-1000 3y ago
  1. DDoS Attacks 2. Port scanning attacks 3. Geolocation attacks (you shouldn't run a server from your home, when your IP Address could be used to find you; be it from a hack or an abuse of the legal process) 4. The US internet relying on a backbone mostly operated by 4 companies (meaning a direct connection is a massive perk) For as much as we moan about Amazon and Cloudflare centralizing things; believe me, just taking out Level 3 / Lumen would be almost an act of war and would affect every ISP and cloud provider. If you took out Lumen, Verizon Enterprise Solutions, and AT&T, just those 3 companies; that's 2.35 million kilometers of fiber going dark. For reference, Comcast has "only" 150,000 km.
  - djbusby 3y ago
    Those are problem and symptoms - not design flaws. How could one design a system that doesn't let DDoS exist on the network?
  - silisili 3y ago
    I actually have a lot of respect for the big backbones. If Goog/MS/Amz owned them all, they'd find some way to wall it off from everyone else. If Comcast or Charter owned them all, our internet would cost about $1000 a month by now. All things considered, IMO they've been fairly good and fair stewards.
justin_oaks 3y ago
I'd appreciate if you'd elaborate on that. (Edit: I see you elaborated somewhat in another comment posted before I finished this comment) I tend to think centralization is more about people's tendencies than it is internet design. As I see it, centralization on the internet happens as the emergent behavior of all of the elements in the system and the forces that exist. In other words, it's the natural result of incentives. People use centralized services because - prices (economies of scale) - network effects ("I can't certain people without WhatsApp") - social pressures ("People look at me weird when I say I don't have an Instagram") - visibility ("There are other CRMs than SalesForce?") - making choices easy ("I don't want to choose between all these options. I'll pick the most familiar") - safety ("Nobody got fired for choosing IBM") - convenience ("It's a pain to self-host") - security ("Google probably protects data better than I can") - etc

deleted 3y ago

[deleted]

SheinhardtWigCo 3y ago

The current situation is preferable to the before times, where any website could be nuked from orbit by anyone with a mild grievance. Cloudflare has succeeded in effectively shutting down that attack vector, and that’s worth the risk of occasional brief but broad outages, at least to me.

dijit 3y ago
There were mitigations for this but it was usually on “IT”s budget and IT was considered a cost center. Really the best innovation in the last 10 years has been the penchant for putting things on development budgets instead which loosened the purse strings considerably. I wish I had sufficient data to back this up but for context; I was part of a small infrastructure team running a little over 1% of all web traffic at one point in a pair of colos (with DDoS protection) for less than £5MM on infra costs. Now I can easily be in the 6 digits with just a CRUD app

rafaelmn 3y ago

If you think internet infrastructure is centralized, boy are you going to be depressed about AI powered services. Those are much more volatile (compute heavy and not enough compute to meet peaks), harder to setup/maintain and benefit greatly from sharing infrastructure for hardware utilization. Everything is moving in that direction.

pvg 3y ago

the inevitable It's readily evitable, just choose not to write the generic, repetitive thing you already know is generic and repetitive.

solardev 3y ago

Is it really that bad? Services go down for a few minutes, maybe a few hours? A thousand other companies are down at the same time. Grab lunch, come back, and it'll have fixed itself. Nobody blames you because it's just another provider outage -- whether it's Cloudflare or AWS or Microsoft or Gmail doesn't really matter, it's a convenient outsourcing of not just network admin but also blame. The alternative is, what, doing it all in-house with a bunch of people on pager duty? Few companies can afford that. Diversifying across multiple cloud providers? Sure, if multi-nines uptime is really critical for your business. For most small-med businesses, it's more valuable to be able to rapidly build on top of existing, professionally-maintained infrastructure in order to deliver some unique business value on top of it, than to recreate that basic network infra themselves. If you're not a FAANG, going down for a few hours a year isn't really a big deal. Especially when you can just blame it on an overall "internet outage at some big provider" rather than some internal fuck-up.

csomar 3y ago
I don't think the problem is a few minutes of down time but all services going down at the same time. You can't order food (Uber down), your email down, your TV down, maybe even the traffic control?
- solardev 3y ago
  Ok, and...? (Edit: Not to be snarky, but really, so what if that happens? Same thing happens if there's a power outage in your neighborhood. Is it a big deal?)
  - king_phil 3y ago
    Are power outages even a thing? My PC at home (germany) is always on, so I'd notice, and I'm involved with energy at our data center (Frankfurt, Germany) and home and external data center had no outage for the last 20 years, at a minimum.
    - solardev 3y ago

cj 3y ago

I think the solution here isn’t “don’t use Cloudflare” … IMO if you need high availability and can’t tolerate downtime, then deploy and split load between 2 or more providers (e.g. Cloudflare, Fastly) and 2 or more clouds. The centralization around AWS and GCP and Azure is more concerning to me than the centralization around Cloudflare. It’s not obvious to me why lots of people using Cloudflare is any worse than lots of people using AWS. If we go even deeper, the entire internet runs on a limited number of undersea cables connecting continents. That in particular has always seemed like the most vulnerable aspect of the internet, considering a targeted attack on < 100 locations could meaningfully cut off internet access for millions (billions?) of people for an extended period of time.

jabart 3y ago
This. If downtime is that big of an issue, you should be on Cloudflare's business/enterprise plan which allows CNAME DNS, have redundant DNS providers with 30s TTLs that flip over to another provider. Better yet if you can 50/50 the traffic and 100/0 it when something goes bad. Downside is cost and DNSSEC has an issue with multiple keys. I think there was an RFC on that though I'm not sure it's well supported yet.
pixl97 3y ago
>then deploy and split load between 2 or more providers (e.g. Cloudflare, Fastly) How does this work when most edge providers manage your DNS to point at the available servers they have? Also, if you "HAVE" to be online, then you're the type of target that's at big risk of DDOS which complicates this even further.
- wbl 3y ago
  Usually each one will give you a CBAME, your DNS or Cedexic decides, sometimes per client. It's a pretty common setup although can complicate config changes and taking advantage of new features.

swader999 3y ago

It's nice to at least be considering blast radius in a figurative manner, hope that lasts.