4 ms·

I didn't know so many sites were depending on Fastly. Stack Overflow, GitHub, reddit, ..., even pip is unavailable. My development workflow is completely janked

by barosl 5y ago

I didn't know so many sites were depending on Fastly. Stack Overflow, GitHub, reddit, ..., even pip is unavailable. My development workflow is completely janked up. It is a bit scary that we are putting so many eggs in one basket.

atymic 5y ago
95k websites use them according to BuiltWith, lots of big names there!
- iso1631 5y ago
  The most popular websites are the ones that need to use CDNs. There's about 4 CDNs of note, so when one is down there's a massive knockon. At work we have fastly, cloudflare, and some others. Fastly has been removed from the pool so we're back up. Fortunatly (or rather thanks to planning) we didn't rely on a cloud service to do that though (say a password stored in cloud based bitwarden)
Nursie 5y ago
Terraform registry too AFAICT.
toyg 5y ago
ah yes, this explains why XKCD is also down.
- b0rsuk 5y ago
  And Noita wiki!
iKevinShah 5y ago
> . It is a bit scary that we are putting so many eggs in one basket. We said the same when AWS was down. Same when fastly CDN is down. It turns out we are not putting eggs in one basket but we are putting all our (an organization's offerings) in one basket (cloud provider). That seems to be the core issue IMO.
notanote 5y ago
I use uBlock Origin with default third-party blocking, so I see such trends happening as I browse. The strong consolidation on fastly feels like a recent development (last year, at a guess). Before that it was edgekey for a while. Maybe we need a crawler that keeps track of the hidden centralisation of the web. Or is someone doing that already? Cloudflare, AWS, fastly, etc. It’s not quite all eggs in one basket, but the amount of baskets is small, and the amount of eggs in them is growing. This is a timely reminder of that.
jamaicahest 5y ago
A client of mine once told me "We are self-hosting our CI/CD servers because when problems occur, I can tell my employees to stay at work until it is fixed. I cannot tell GitHub to stay at work until they fix their issue."
- lxgr 5y ago
  That seems like a weird way of measuring success (person-hours rather than some goal/outcome-oriented metrics like uptime).
  - regularfry 5y ago
    Not only that, it's just odd logic. I'm going to prevent my employees from domain-specific value-add by forcing them to overwork on generic tooling? What?
  - sidlls 5y ago
    I don't think that's what was intended. It reads to me there's more perception of control (employees directed to work) to achieve the goal (restoring service) during an outage. Outages happen regardless of uptime guarantees--it's just a matter of how often and how long they last.
    - lxgr 5y ago
      That's how I understood it as well, and it's a pattern I've seen in (bad) decision making: While a stakeholder might feel more "in control" by being able to look over the shoulder of an on-call engineer, asking somebody to reboot the database, etc., this very often does not translate into higher performance metrics. Of course, the opposite would be outsourcing all responsibilities, so there obviously is an art to balancing these two extremes.

by barosl 5y ago

atymic 5y ago

95k websites use them according to BuiltWith, lots of big names there!

iso1631 5y ago
The most popular websites are the ones that need to use CDNs. There's about 4 CDNs of note, so when one is down there's a massive knockon. At work we have fastly, cloudflare, and some others. Fastly has been removed from the pool so we're back up. Fortunatly (or rather thanks to planning) we didn't rely on a cloud service to do that though (say a password stored in cloud based bitwarden)

Nursie 5y ago

Terraform registry too AFAICT.

toyg 5y ago

ah yes, this explains why XKCD is also down.

b0rsuk 5y ago
And Noita wiki!

iKevinShah 5y ago

> . It is a bit scary that we are putting so many eggs in one basket. We said the same when AWS was down. Same when fastly CDN is down. It turns out we are not putting eggs in one basket but we are putting all our (an organization's offerings) in one basket (cloud provider). That seems to be the core issue IMO.

notanote 5y ago

I use uBlock Origin with default third-party blocking, so I see such trends happening as I browse. The strong consolidation on fastly feels like a recent development (last year, at a guess). Before that it was edgekey for a while. Maybe we need a crawler that keeps track of the hidden centralisation of the web. Or is someone doing that already? Cloudflare, AWS, fastly, etc. It’s not quite all eggs in one basket, but the amount of baskets is small, and the amount of eggs in them is growing. This is a timely reminder of that.

jamaicahest 5y ago

A client of mine once told me "We are self-hosting our CI/CD servers because when problems occur, I can tell my employees to stay at work until it is fixed. I cannot tell GitHub to stay at work until they fix their issue."

lxgr 5y ago
That seems like a weird way of measuring success (person-hours rather than some goal/outcome-oriented metrics like uptime).
- regularfry 5y ago
  Not only that, it's just odd logic. I'm going to prevent my employees from domain-specific value-add by forcing them to overwork on generic tooling? What?
- sidlls 5y ago
  I don't think that's what was intended. It reads to me there's more perception of control (employees directed to work) to achieve the goal (restoring service) during an outage. Outages happen regardless of uptime guarantees--it's just a matter of how often and how long they last.
  - lxgr 5y ago
    That's how I understood it as well, and it's a pattern I've seen in (bad) decision making: While a stakeholder might feel more "in control" by being able to look over the shoulder of an on-call engineer, asking somebody to reboot the database, etc., this very often does not translate into higher performance metrics. Of course, the opposite would be outsourcing all responsibilities, so there obviously is an art to balancing these two extremes.