3 ms·

> commitment to ensure that customer configurations cannot break the whole system You can't just ensure a config change won't break things in large distributed

> commitment to ensure that customer configurations cannot break the whole system You can't just ensure a config change won't break things in large distributed systems, it's too complex with too many factors, there will always be risk. To mitigate your risk, youd want to design your system to do progressive, regional rollouts, with canaries to attempt to detect and isolate before a wide spread outage occurs. Even if you have all of this set up, there is still risk that your regions and systems are not fully isolated and outages could cascade anyway. There will always be risk, there will always be errors. This is why SLAs and SLOs exist, they define and codify an agreement of what an outage is and what compensation is required if the agreement isn't met. You can read Fastlys SLA here: https://docs.fastly.com/products/service-availability-sla https://docs.fastly.com/products/service-availability-sla

fafner 5y ago
Unfortunately some customer changes will need to go out quickly and globally for a CDN. Going much slower might not be a good option. Canarying should detect this. Not clear if they do this or the canary failed to report this. Sharding by customers could help reduce blast radius. But maybe not by much of this was a very big customer.
- thethethethe 5y ago
  > Unfortunately some customer changes will need to go out quickly and globally for a CDN Why is this the case? I don't have too much knowledge of CDN architecture so I am curious
  - rblatz 5y ago
    Pushing out new versions of your site. You can’t have the new assets on half the nodes that are serving your site otherwise your site goes down while things slowly propagate.
  - notyourday 5y ago
    > Why is this the case? I don't have too much knowledge of CDN architecture so I am curious Fastly is not really a regular CDN. It is a fully programmable edge cache with cache control algorithms decided and controlled by the customer running at the edges. You can think of Fastly configuration as a part of your code base where it is for you to decide if you want to perform the action on the edge on a per-request basis rather than on the origin per cached request basis. That in turn means that if you do deploy to your API/web 50 times a day, you would are likely to deploy your Fastly configurations about the same number of times
- aliasEli 5y ago
  Sharding would also introduce additional complexity into the system, which is also risky.

fafner 5y ago

Unfortunately some customer changes will need to go out quickly and globally for a CDN. Going much slower might not be a good option. Canarying should detect this. Not clear if they do this or the canary failed to report this. Sharding by customers could help reduce blast radius. But maybe not by much of this was a very big customer.

thethethethe 5y ago
> Unfortunately some customer changes will need to go out quickly and globally for a CDN Why is this the case? I don't have too much knowledge of CDN architecture so I am curious
- rblatz 5y ago
  Pushing out new versions of your site. You can’t have the new assets on half the nodes that are serving your site otherwise your site goes down while things slowly propagate.
- notyourday 5y ago
  > Why is this the case? I don't have too much knowledge of CDN architecture so I am curious Fastly is not really a regular CDN. It is a fully programmable edge cache with cache control algorithms decided and controlled by the customer running at the edges. You can think of Fastly configuration as a part of your code base where it is for you to decide if you want to perform the action on the edge on a per-request basis rather than on the origin per cached request basis. That in turn means that if you do deploy to your API/web 50 times a day, you would are likely to deploy your Fastly configurations about the same number of times
aliasEli 5y ago
Sharding would also introduce additional complexity into the system, which is also risky.