5 ms·

I might sound naive, but as a true hyperscale internet company to plan for disaster scenario like this as a consumer of fastly? How could you plan for an outag

I might sound naive, but as a true hyperscale internet company to plan for disaster scenario like this as a consumer of fastly? How could you plan for an outage like this by fastly and how could you mitigate this?

raphaelj 5y ago
Use a short TTL on the CDN subdomain you use. Then setup an alternative CDN provider in advance, so that you can switch from one to the other in a matter of minutes.
- flurdy 5y ago
  Using a short TTL on a DNS entry has performance issues [1]. But a long one also isn't ideal [2]. [1] https://twitter.com/RichardTowers/status/1402312111078903812 https://twitter.com/RichardTowers/status/1402312111078903812 [2] https://twitter.com/RichardTowers/status/1402312308077076484 https://twitter.com/RichardTowers/status/1402312308077076484
efficax 5y ago
Using DNS to failover to another CDN is pretty much the only solution if the solution is not "build out your own global edge infrastructure". I was thinking more about this though and it has its own problems. You want a short TTL so failover is fast, but this increases the number of DNS lookups people have to do (and DNS lookups can be very slow!). Additionally, a short TTL means you're more vulnerable to problems like the dyndns attack [1] from 2016: names with longer TTLs were up for longer since they preserved the correct DNS records for longer. But if you have a long TTL, even if you fail over, you'll still be down for at least as long as the DNS TTL pointing to the bad CDN. Maybe, you could do DNS roundrobin against multiple CDN providers at once. Say you used 4, then if one went down, only 25% of requests would fail, and you could just remove the failing entry. This seems very expensive! Honestly, the cost of these solutions is probably not worth it. The product I work on went partially down during the fastly outage. Then it came back up and everything is back to normal. It really won't impact us much at all. Shrug. 1. https://en.wikipedia.org/wiki/2016_Dyn_cyberattack https://en.wikipedia.org/wiki/2016_Dyn_cyberattack

raphaelj 5y ago

Use a short TTL on the CDN subdomain you use. Then setup an alternative CDN provider in advance, so that you can switch from one to the other in a matter of minutes.

flurdy 5y ago
Using a short TTL on a DNS entry has performance issues [1]. But a long one also isn't ideal [2]. [1] https://twitter.com/RichardTowers/status/1402312111078903812 https://twitter.com/RichardTowers/status/1402312111078903812 [2] https://twitter.com/RichardTowers/status/1402312308077076484 https://twitter.com/RichardTowers/status/1402312308077076484

efficax 5y ago

Using DNS to failover to another CDN is pretty much the only solution if the solution is not "build out your own global edge infrastructure". I was thinking more about this though and it has its own problems. You want a short TTL so failover is fast, but this increases the number of DNS lookups people have to do (and DNS lookups can be very slow!). Additionally, a short TTL means you're more vulnerable to problems like the dyndns attack [1] from 2016: names with longer TTLs were up for longer since they preserved the correct DNS records for longer. But if you have a long TTL, even if you fail over, you'll still be down for at least as long as the DNS TTL pointing to the bad CDN. Maybe, you could do DNS roundrobin against multiple CDN providers at once. Say you used 4, then if one went down, only 25% of requests would fail, and you could just remove the failing entry. This seems very expensive! Honestly, the cost of these solutions is probably not worth it. The product I work on went partially down during the fastly outage. Then it came back up and everything is back to normal. It really won't impact us much at all. Shrug. 1. https://en.wikipedia.org/wiki/2016_Dyn_cyberattack https://en.wikipedia.org/wiki/2016_Dyn_cyberattack