4 ms·

Have they published a post-mortem for the outage yet? I'm curious as to what happened here

takeda 5y ago
They wrote that they had to reinstall Workbench. Isn't that too early for pay mortem? I didn't experience it myself, but I think it happened at most few hours ago.
unityByFreedom 5y ago
Here it is https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage Posted about 17 hours after the incident. In short, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/ https://www.cloudflare.com/learning/security/glossary/what-i...
- maxgashkov 5y ago
  Zero Trust is not really applicable here. ZT is about not implicitly allowing someone elevated access just because they have access to network and requiring explicit grant based on the identity and other rules. Fastly's downtime seems to be caused by an automatically generated config that got deployed in production as a result of change requested by a legitimate customer. This happened to CF in its early days and I really doubt that ZT had anything to do with the fact that they do not have this kind of problem anymore. It's probably some sanity checks before they deploy updated lua scripts to their fleet of nginx's if anything.
- joshuamorton 5y ago
  Zero trust has little to do with individual clients and more to do with the idea of the internal "corp" network not being trusted. That is, there isn't a conventional network you VPN into at which point all traffic is trusted. The generic topic your looking for is probably something like "customer isolation" ("service isolation" might also be relevant, but is used also in the context of "tenant isolation" which isn't really what you want). See this thread: https://news.ycombinator.com/item?id=25237836 https://news.ycombinator.com/item?id=25237836 for some talk about how AWS does "cellularization" which is a form of workload/service isolation/partitioning. In general I don't think there's much discussion of this issue on the wider web.

They wrote that they had to reinstall Workbench. Isn't that too early for pay mortem? I didn't experience it myself, but I think it happened at most few hours ago.

unityByFreedom 5y ago

Here it is https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage Posted about 17 hours after the incident. In short, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/ https://www.cloudflare.com/learning/security/glossary/what-i...

maxgashkov 5y ago
Zero Trust is not really applicable here. ZT is about not implicitly allowing someone elevated access just because they have access to network and requiring explicit grant based on the identity and other rules. Fastly's downtime seems to be caused by an automatically generated config that got deployed in production as a result of change requested by a legitimate customer. This happened to CF in its early days and I really doubt that ZT had anything to do with the fact that they do not have this kind of problem anymore. It's probably some sanity checks before they deploy updated lua scripts to their fleet of nginx's if anything.
joshuamorton 5y ago
Zero trust has little to do with individual clients and more to do with the idea of the internal "corp" network not being trusted. That is, there isn't a conventional network you VPN into at which point all traffic is trusted. The generic topic your looking for is probably something like "customer isolation" ("service isolation" might also be relevant, but is used also in the context of "tenant isolation" which isn't really what you want). See this thread: https://news.ycombinator.com/item?id=25237836 https://news.ycombinator.com/item?id=25237836 for some talk about how AWS does "cellularization" which is a form of workload/service isolation/partitioning. In general I don't think there's much discussion of this issue on the wider web.