4 ms·

So, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot b

So, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/ https://www.cloudflare.com/learning/security/glossary/what-i...

bombcar 5y ago
Yeah it seems an architecture that even allows a single end-customer issue to take down the entire network may need a bit of rearchitecting.
darwinwhy 5y ago
Why do so many big companies use Fastly when Cloudflare (from the outside, as someone who doesn't know much about the space) looks to be so much cleaner and more technically sophisticated? Am I being brainwashed by their blog posts?
- wjossey 5y ago
  Truly different capabilities under the hood. Yes, they are both CDN providers, but fastly offers a remarkable amount of customization that cloudflare does not. For 99% of customers, one can argue that cloudflare is more than sufficient. For 1% of customers, fastly is arguably the correct choice just based on feature set alone. So, in summary, you can certainly compare the two, but for certain customers cloudflare lacks the feature set they may choose to use on fastly.
  - ascorbic 5y ago
    Yes, to the extent that Fastly lets you upload your own VCL configuration files. This was the source of the problem here, but is incredibly powerful for complex use cases.
- lmm 5y ago
  I don't know about you but I find the prospect of a Cloudflare monoculture pretty worrying, especially since they've already demonstrate a willingness to kick off users they don't like. (I also think the https veneer that they offer is misleading to end users and bad for everyone on the internet, though not everyone will agree with that).
  - KirillPanov 5y ago
    I too think Cloudflare's "reverse HTTPS proxy" approach (where they have a CA-signed certificate for every domain that gets pointed at them) is bad for the Internet. But how does Fastly avoid this problem? It's really more a symptom of the "web pki" trainwreck than anything else. I tried looking on Fastly's website for technical details, but like every other corporate website it was an impenetrable mass of marketing bling and partner logos.
    - lmm 5y ago
      As much as that's not ideal, if the proxy is actually using HTTPS and verifying the upstream certificate then I don't think it breaks the user's security expectations too badly. But CloudFlare also offer a mode where they will serve HTTPS to the user but connect to your upstream via unencrypted HTTP over the public internet, which I think is just shockingly awful compared to what a user expects a site that uses HTTPS to do.
fafner 5y ago
Zero trust seems to be very unrelated to this issue. The issue seems to have been a poison config breaking fastly stack. Zero trust is about verifying authentication of devices/users. Unrelated things really.
- unityByFreedom 5y ago
  It's much more than that. It's a whole approach to architecture. The post I linked says it best.
  - fafner 5y ago
    Authentication/trust is simply unrelated to this outage. The post you linked seems to have no relevance.
    - unityByFreedom 5y ago
      One of Cloudflare's top engineers previously wrote in this forum, > This incident emphasizes the importance of the Zero Trust model that Cloudflare follows and provides to customers, which ensures that if any one system or vendor is compromised, it does not compromise the entire organization. [1] Authentication is a part of a zero-trust model, not the whole thing. > No single specific technology is associated with zero trust architecture; it is a holistic approach to network security that incorporates several different principles and technologies. [2] [1] https://news.ycombinator.com/item?id=26407364 https://news.ycombinator.com/item?id=26407364 [2] https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/ https://www.cloudflare.com/learning/security/glossary/what-i...
      - johncolanduoni 5y ago
        They were referring to a completely different incident, involving compromised authentication to a camera system. I’d love to hear an explanation of how a zero-trust model would apply to this situation with Fastly. Seems like it would have to apply to a lot of multi-tenant resource exhaustion issues since we know so little about the specifics on the Fastly incident.
thethethethe 5y ago
> commitment to ensure that customer configurations cannot break the whole system You can't just ensure a config change won't break things in large distributed systems, it's too complex with too many factors, there will always be risk. To mitigate your risk, youd want to design your system to do progressive, regional rollouts, with canaries to attempt to detect and isolate before a wide spread outage occurs. Even if you have all of this set up, there is still risk that your regions and systems are not fully isolated and outages could cascade anyway. There will always be risk, there will always be errors. This is why SLAs and SLOs exist, they define and codify an agreement of what an outage is and what compensation is required if the agreement isn't met. You can read Fastlys SLA here: https://docs.fastly.com/products/service-availability-sla https://docs.fastly.com/products/service-availability-sla
- fafner 5y ago
  Unfortunately some customer changes will need to go out quickly and globally for a CDN. Going much slower might not be a good option. Canarying should detect this. Not clear if they do this or the canary failed to report this. Sharding by customers could help reduce blast radius. But maybe not by much of this was a very big customer.
  - thethethethe 5y ago
    > Unfortunately some customer changes will need to go out quickly and globally for a CDN Why is this the case? I don't have too much knowledge of CDN architecture so I am curious
    - rblatz 5y ago
      Pushing out new versions of your site. You can’t have the new assets on half the nodes that are serving your site otherwise your site goes down while things slowly propagate.
    - notyourday 5y ago
      > Why is this the case? I don't have too much knowledge of CDN architecture so I am curious Fastly is not really a regular CDN. It is a fully programmable edge cache with cache control algorithms decided and controlled by the customer running at the edges. You can think of Fastly configuration as a part of your code base where it is for you to decide if you want to perform the action on the edge on a per-request basis rather than on the origin per cached request basis. That in turn means that if you do deploy to your API/web 50 times a day, you would are likely to deploy your Fastly configurations about the same number of times
notyourday 5y ago
Fastly and Cloudflare use a totally different approaches to edge control. If Fastly implements Cloudflare approach its key advantage would be gone.

bombcar 5y ago

Yeah it seems an architecture that even allows a single end-customer issue to take down the entire network may need a bit of rearchitecting.

darwinwhy 5y ago

Why do so many big companies use Fastly when Cloudflare (from the outside, as someone who doesn't know much about the space) looks to be so much cleaner and more technically sophisticated? Am I being brainwashed by their blog posts?

wjossey 5y ago
Truly different capabilities under the hood. Yes, they are both CDN providers, but fastly offers a remarkable amount of customization that cloudflare does not. For 99% of customers, one can argue that cloudflare is more than sufficient. For 1% of customers, fastly is arguably the correct choice just based on feature set alone. So, in summary, you can certainly compare the two, but for certain customers cloudflare lacks the feature set they may choose to use on fastly.
- ascorbic 5y ago
  Yes, to the extent that Fastly lets you upload your own VCL configuration files. This was the source of the problem here, but is incredibly powerful for complex use cases.
lmm 5y ago
I don't know about you but I find the prospect of a Cloudflare monoculture pretty worrying, especially since they've already demonstrate a willingness to kick off users they don't like. (I also think the https veneer that they offer is misleading to end users and bad for everyone on the internet, though not everyone will agree with that).
- KirillPanov 5y ago
  I too think Cloudflare's "reverse HTTPS proxy" approach (where they have a CA-signed certificate for every domain that gets pointed at them) is bad for the Internet. But how does Fastly avoid this problem? It's really more a symptom of the "web pki" trainwreck than anything else. I tried looking on Fastly's website for technical details, but like every other corporate website it was an impenetrable mass of marketing bling and partner logos.
  - lmm 5y ago
    As much as that's not ideal, if the proxy is actually using HTTPS and verifying the upstream certificate then I don't think it breaks the user's security expectations too badly. But CloudFlare also offer a mode where they will serve HTTPS to the user but connect to your upstream via unencrypted HTTP over the public internet, which I think is just shockingly awful compared to what a user expects a site that uses HTTPS to do.

fafner 5y ago

Zero trust seems to be very unrelated to this issue. The issue seems to have been a poison config breaking fastly stack. Zero trust is about verifying authentication of devices/users. Unrelated things really.

unityByFreedom 5y ago
It's much more than that. It's a whole approach to architecture. The post I linked says it best.
- fafner 5y ago
  Authentication/trust is simply unrelated to this outage. The post you linked seems to have no relevance.
  - unityByFreedom 5y ago
    One of Cloudflare's top engineers previously wrote in this forum, > This incident emphasizes the importance of the Zero Trust model that Cloudflare follows and provides to customers, which ensures that if any one system or vendor is compromised, it does not compromise the entire organization. [1] Authentication is a part of a zero-trust model, not the whole thing. > No single specific technology is associated with zero trust architecture; it is a holistic approach to network security that incorporates several different principles and technologies. [2] [1] https://news.ycombinator.com/item?id=26407364 https://news.ycombinator.com/item?id=26407364 [2] https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/ https://www.cloudflare.com/learning/security/glossary/what-i...
    - johncolanduoni 5y ago
      They were referring to a completely different incident, involving compromised authentication to a camera system. I’d love to hear an explanation of how a zero-trust model would apply to this situation with Fastly. Seems like it would have to apply to a lot of multi-tenant resource exhaustion issues since we know so little about the specifics on the Fastly incident.

thethethethe 5y ago

> commitment to ensure that customer configurations cannot break the whole system You can't just ensure a config change won't break things in large distributed systems, it's too complex with too many factors, there will always be risk. To mitigate your risk, youd want to design your system to do progressive, regional rollouts, with canaries to attempt to detect and isolate before a wide spread outage occurs. Even if you have all of this set up, there is still risk that your regions and systems are not fully isolated and outages could cascade anyway. There will always be risk, there will always be errors. This is why SLAs and SLOs exist, they define and codify an agreement of what an outage is and what compensation is required if the agreement isn't met. You can read Fastlys SLA here: https://docs.fastly.com/products/service-availability-sla https://docs.fastly.com/products/service-availability-sla

fafner 5y ago
Unfortunately some customer changes will need to go out quickly and globally for a CDN. Going much slower might not be a good option. Canarying should detect this. Not clear if they do this or the canary failed to report this. Sharding by customers could help reduce blast radius. But maybe not by much of this was a very big customer.
- thethethethe 5y ago
  > Unfortunately some customer changes will need to go out quickly and globally for a CDN Why is this the case? I don't have too much knowledge of CDN architecture so I am curious
  - rblatz 5y ago
    Pushing out new versions of your site. You can’t have the new assets on half the nodes that are serving your site otherwise your site goes down while things slowly propagate.
  - notyourday 5y ago
    > Why is this the case? I don't have too much knowledge of CDN architecture so I am curious Fastly is not really a regular CDN. It is a fully programmable edge cache with cache control algorithms decided and controlled by the customer running at the edges. You can think of Fastly configuration as a part of your code base where it is for you to decide if you want to perform the action on the edge on a per-request basis rather than on the origin per cached request basis. That in turn means that if you do deploy to your API/web 50 times a day, you would are likely to deploy your Fastly configurations about the same number of times

notyourday 5y ago

Fastly and Cloudflare use a totally different approaches to edge control. If Fastly implements Cloudflare approach its key advantage would be gone.