4 ms·

fta: "The outage has raised questions about relying on a handful of companies to run the vast infrastructure that underpins the internet." I read that phrase e

fta: "The outage has raised questions about relying on a handful of companies to run the vast infrastructure that underpins the internet." I read that phrase everytime something like this happens and yet we all still rely on the same handful of companies.

axiosgunnar 5y ago
Would a smaller CDN provider have no outages at all?
- sgtfrankieboy 5y ago
  No, but it would take down less sites if it does have one.
  - gpm 5y ago
    Is that a feature? If we have 12 small cdns have 12 outages in a year (combined, 1/year each), each time bringing down 10,000 websites, is that better than 1 large cdn having one outage during the year bringing down 120,000 websites? If I'm the website owner I think I prefer the latter, my customers blame the cdn instead of me. If I'm the cdn owner I definitely prefer the latter, more customers to amortize my costs over.
    - iso1210 5y ago
      From a customer point of view diversity is far better it's great - if Sainsburys is closed for some reason I'll go to Tesco. Certainly don't want a situation where all the shops are closed.
- martius 5y ago
  It may be more acceptable to have more frequent outages if the impact radius (number of websites or services impacted) is smaller.
- zelon88 5y ago
  Obviously that isn't an honest comparison. If you're asking whether or not 10 small CDN providers could provide a more robust, higher quality service with more uptime than one large CDN provider, then I think the answer is probably "yes."
- unhammer 5y ago
  There's kind of a race-to-the-bottom[0] wrt. decentralisation. It'd be better for the internet as a whole if we don't always pick the most popular (so when your email's CDN goes down you can still communicate on chat, when CNN goes down you can still read BBC). But as an individual I have strong incentives to pick the one everyone else picks, because that's presumably the most stable/documented/lowest cost due to volume. [0] https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
notacoward 5y ago
It's the tech equivalent of "thoughts and prayers" isn't it?
adrr 5y ago
There are at least 5 major CDNs out there. CloudFront, Fastly,Cloudflare, Akamai, Google CDN. You can use more than just 1. Shopify uses two. Akamai and Fastly.
- viraptor 5y ago
  If you use them as pure file-serving CDN, then sure. But once you start adding extra logic, headers, routing, etc. the features don't fully align. Or you need to keep to the minimum common featureset.
- 0xcoffee 5y ago
  Does anyone know if Azure/Google/Amazon can provide some 'multi-cdn' setup out of the box? The way to change these points of failure is for the big boys to change their defaults.
  - adrr 5y ago
    Do it at the DNS layer. Route53 has failover support out of the box that should work for it. You can setup a monitor and it will switch dns entries on a failure.
- tyingq 5y ago
  It is getting increasing tricky to have enough redundancy at a basic level to avoid a major player outage from affecting you. For example, you would probably want at least one authoritative DNS server that isn't either of your CDN providers. And knowing some details about how these players sometimes use each other, like that Google's Firebase uses Fastly.
  - gonzo41 5y ago
    Are you bigger than a major player is the question I'd be asking. Maybe risking it is fine.
    - tyingq 5y ago
politelemon 5y ago
I think it will continue to happen. People and orgs, when picking a service, don't have an incentive or a way to ask around to see what percentage of similar-service users are using that particular service. People and orgs will always flow towards cheap/popular/well-known services. On the other hand, those handful of companies could be asked to structure their services so that an outage only affects a portion of customers and not all their customers. However! That would be more inefficient for them, and more expensive, and that cost would cause the people and orgs mentioned earlier to just flow towards the company that took those shortcuts.
- vikramkr 5y ago
  There is an incentive to see what everyone else is using, so that you can use it too. Choose boring technologies applies to infra as well. Go with a big stable well known can or cloud provider instead of trusting your service to some fly by night startup etc.
perlgeek 5y ago
That's because many such business have grown so large because they benefit from scale. Some examples: Social network: you only engage on one if your friends/family/coworkers are on the same network. Search engine: needs to index "the whole Internet", which is less expensive per user if you have more users CDN: works best if you have edge nodes everywhere, which is quite capital intensive, which is why you need many customers to distribute it over. ... and so on. We might not like it, but many of these quasi monopolies are based on fundamental economics, not (just) on the greed of the companies.
- ryukafalz 5y ago
  > Social network: you only engage on one if your friends/family/coworkers are on the same network. This one isn't exactly based on fundamental economics given that federated social networks exist. Email has similar network effects and is not centralized.
  - toxik 5y ago
    I think this argument could be made for many of these.
  - perlgeek 5y ago
    None of the federated social networks seems to have reached the scale of the biggest centralized social networks. Which leads me to believe that economics and incentives favor big, centralized social networks.
kleinsch 5y ago
If you're making a purchasing recommendation for your company, do you want to tell your boss that you're recommending not going with your best option, or even 2nd, but that your company should use the 4th or 5th best CDN as a way to diversify the Internet? Seems pretty altruistic, but not a great way to keep your job.
JulianMorrison 5y ago
The answer is that before the existence of those companies to rely upon, people didn't rely upon them, they just accepted the lag, or they hand hacked the same globally distributed approach on their own, and it sucked for them, and it wasn't too great for users either. CDNs are big because that's what their function is, to reach the world and absorb traffic spikes, and take the complicated business of distribution of edge servers out of the hands of the people who just want to run a website. The trade off here is intrinsic and accepting the risks of big CDNs is the right answer.
- mathattack 5y ago
  Exactly. The question isn’t “a few CDNs” vs “Many CDNs”. There are too many economies of scale. It’s really CDN vs Not? (Roll your own just isn’t feasible except for a half dozen of the largest tech firms)
- TheDudeMan 5y ago
  Well, you could use two CDNs instead of just one. But that costs money.
toyg 5y ago
One of the side-effects of the efficiency obsession that capitalism generates, is the constant strive for centralization. We took a technology stack designed to survive nuclear attacks, and turned it into something where a single bug can take down half the services on it. Why? Because on the flip side, a single improvement in the centralized service can automatically cascade to all the businesses using it. Efficiency is a double-edged sword.

axiosgunnar 5y ago

Would a smaller CDN provider have no outages at all?

sgtfrankieboy 5y ago
No, but it would take down less sites if it does have one.
- gpm 5y ago
  Is that a feature? If we have 12 small cdns have 12 outages in a year (combined, 1/year each), each time bringing down 10,000 websites, is that better than 1 large cdn having one outage during the year bringing down 120,000 websites? If I'm the website owner I think I prefer the latter, my customers blame the cdn instead of me. If I'm the cdn owner I definitely prefer the latter, more customers to amortize my costs over.
  - iso1210 5y ago
    From a customer point of view diversity is far better it's great - if Sainsburys is closed for some reason I'll go to Tesco. Certainly don't want a situation where all the shops are closed.
martius 5y ago
It may be more acceptable to have more frequent outages if the impact radius (number of websites or services impacted) is smaller.
zelon88 5y ago
Obviously that isn't an honest comparison. If you're asking whether or not 10 small CDN providers could provide a more robust, higher quality service with more uptime than one large CDN provider, then I think the answer is probably "yes."
unhammer 5y ago
There's kind of a race-to-the-bottom[0] wrt. decentralisation. It'd be better for the internet as a whole if we don't always pick the most popular (so when your email's CDN goes down you can still communicate on chat, when CNN goes down you can still read BBC). But as an individual I have strong incentives to pick the one everyone else picks, because that's presumably the most stable/documented/lowest cost due to volume. [0] https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

notacoward 5y ago

It's the tech equivalent of "thoughts and prayers" isn't it?

adrr 5y ago

There are at least 5 major CDNs out there. CloudFront, Fastly,Cloudflare, Akamai, Google CDN. You can use more than just 1. Shopify uses two. Akamai and Fastly.

viraptor 5y ago
If you use them as pure file-serving CDN, then sure. But once you start adding extra logic, headers, routing, etc. the features don't fully align. Or you need to keep to the minimum common featureset.
0xcoffee 5y ago
Does anyone know if Azure/Google/Amazon can provide some 'multi-cdn' setup out of the box? The way to change these points of failure is for the big boys to change their defaults.
- adrr 5y ago
  Do it at the DNS layer. Route53 has failover support out of the box that should work for it. You can setup a monitor and it will switch dns entries on a failure.
tyingq 5y ago
It is getting increasing tricky to have enough redundancy at a basic level to avoid a major player outage from affecting you. For example, you would probably want at least one authoritative DNS server that isn't either of your CDN providers. And knowing some details about how these players sometimes use each other, like that Google's Firebase uses Fastly.
- gonzo41 5y ago
  Are you bigger than a major player is the question I'd be asking. Maybe risking it is fine.
  - tyingq 5y ago

politelemon 5y ago

I think it will continue to happen. People and orgs, when picking a service, don't have an incentive or a way to ask around to see what percentage of similar-service users are using that particular service. People and orgs will always flow towards cheap/popular/well-known services. On the other hand, those handful of companies could be asked to structure their services so that an outage only affects a portion of customers and not all their customers. However! That would be more inefficient for them, and more expensive, and that cost would cause the people and orgs mentioned earlier to just flow towards the company that took those shortcuts.

vikramkr 5y ago
There is an incentive to see what everyone else is using, so that you can use it too. Choose boring technologies applies to infra as well. Go with a big stable well known can or cloud provider instead of trusting your service to some fly by night startup etc.

perlgeek 5y ago

That's because many such business have grown so large because they benefit from scale. Some examples: Social network: you only engage on one if your friends/family/coworkers are on the same network. Search engine: needs to index "the whole Internet", which is less expensive per user if you have more users CDN: works best if you have edge nodes everywhere, which is quite capital intensive, which is why you need many customers to distribute it over. ... and so on. We might not like it, but many of these quasi monopolies are based on fundamental economics, not (just) on the greed of the companies.

ryukafalz 5y ago
> Social network: you only engage on one if your friends/family/coworkers are on the same network. This one isn't exactly based on fundamental economics given that federated social networks exist. Email has similar network effects and is not centralized.
- toxik 5y ago
  I think this argument could be made for many of these.
- perlgeek 5y ago
  None of the federated social networks seems to have reached the scale of the biggest centralized social networks. Which leads me to believe that economics and incentives favor big, centralized social networks.

kleinsch 5y ago

If you're making a purchasing recommendation for your company, do you want to tell your boss that you're recommending not going with your best option, or even 2nd, but that your company should use the 4th or 5th best CDN as a way to diversify the Internet? Seems pretty altruistic, but not a great way to keep your job.

JulianMorrison 5y ago

The answer is that before the existence of those companies to rely upon, people didn't rely upon them, they just accepted the lag, or they hand hacked the same globally distributed approach on their own, and it sucked for them, and it wasn't too great for users either. CDNs are big because that's what their function is, to reach the world and absorb traffic spikes, and take the complicated business of distribution of edge servers out of the hands of the people who just want to run a website. The trade off here is intrinsic and accepting the risks of big CDNs is the right answer.

mathattack 5y ago
Exactly. The question isn’t “a few CDNs” vs “Many CDNs”. There are too many economies of scale. It’s really CDN vs Not? (Roll your own just isn’t feasible except for a half dozen of the largest tech firms)
TheDudeMan 5y ago
Well, you could use two CDNs instead of just one. But that costs money.

toyg 5y ago

One of the side-effects of the efficiency obsession that capitalism generates, is the constant strive for centralization. We took a technology stack designed to survive nuclear attacks, and turned it into something where a single bug can take down half the services on it. Why? Because on the flip side, a single improvement in the centralized service can automatically cascade to all the businesses using it. Efficiency is a double-edged sword.