3 ms·

Not in the article: how one Fastly customer triggered the bug. Just a quote and a promise that an RCA will be posted.

tyingq 5y ago
Fastly doesn't appear to be sharing that detail. Their own blog post is similarly vague about the exact cause. https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage Edit: That blog post does say this: "On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances." The scheduled maintenance on May 12 was this: https://status.fastly.com/incidents/dlsphjqst537 https://status.fastly.com/incidents/dlsphjqst537 Based on that, it sounds like maybe a configuration change could deploy new cache nodes with ip addresses that a customer hasn't explicitly allowed to talk to their backend: "When this change is applied, customers may observe additional origin traffic as new cache nodes retrieve content from origin. Please be sure to check that your origin access lists allow the full range of Fastly IP addresses"
- abluecloud 5y ago
  Why is everyone banging on about this. It's a blog post from the same day, a decent post mortem takes a while to put together and assuming the bug isn't fully patched across their entire CDN, why would they post the information.
  - tyingq 5y ago
    I wasn't "banging on", I was answering why the article didn't mention the cause...because the source didn't either.
  - notacoward 5y ago
    Everyone is "banging on" because there are important lessons to be learned from such incidents, and people want to learn. They hunger for more details about the generalizable aspects of the bug, even if a full post mortem that also covers internal processes etc. might take longer to do. Having participated in many post mortems, in many roles, for systems just as complex, I believe it's entirely possible to provide that information the next (not same) day. Is that still setting the bar too high? Perhaps. Fastly deserves kudos for providing even the level of information that they have, since that's already above the pathetic industry standard, but I don't think there's anything bad about wanting more. Defensiveness is the enemy of effective post mortems.
  - jffry 5y ago
    Plus it would be weird to present just that specific information, outside of the context of a post mortem / failure chain analysis type discussion.
    - tyingq 5y ago
      That's true, though they are also saying things like "We created a permanent fix for the bug and began deploying it at 17:25.". "Permanent fix" sort of implies they understood the issue really well.
      - jffry 5y ago
        That's my point though. Even though they may understand the immediate flaw in their code that caused the issue, there's not much use (for them or their customers) in just talking in detail about that specific flaw. I'd go so far as to argue that the specifics of the flaw are immaterial right now. At this stage, the important thing is that they have identified a specific code change that was the proximate cause of the issue, and have a mitigation in place. This is contrasted with more mysterious and hard-to-track-down failures. ("We are working to understand why our systems are down and will post another update in 30 minutes") What will take time, and the thing which will be interesting, is failure tree analysis. (You might hear the phrase "failure chain" or "root cause" but IMO it's quite rare for things to be so linear). That can help identify opportunities to improve processes at many different levels of the product lifecycle. Humans are fallible, and there's no way we can write bug-free software, so the solution has to be more robust than "hope that every member of our organization never makes a mistake again"
        tyingq 5y ago
        Yes, I was saying I would have avoided words like "permanent fix", because it sets unrealistic expectations.
- addingnumbers 5y ago
  The paragraph you quoted is just describing a side effect of adding nodes. That excerpt appears in every one of their capacity expansion announcements, going back years.
  - tyingq 5y ago
    Ah, interesting. Though the change itself reads like it was just adding capacity in one location (Newark). I don't see any other changes mentioned for that date.
Dobbs 5y ago
My uneducated hypothesis is that Fastly runs varnish. Presumably they have some process that collects data from their config system generates the VCL (varnishes psuedo-C configuration language which compiles directly to C). Somehow a customer configured something in such a way that it generated a bad VCL file which then caused it to lose all configuration, or caused one domain to incorrectly garble up traffic for all domains. I can poke plenty of holes in this hypothesis, like fastly likely not deploying configuration to all nodes but only subsets. Looking forward to the deeper post.
- tyingq 5y ago
  It's interesting to me that they still run Varnish, since it doesn't have https/tls built-in. I do get that VCL is more expressive than similar capabilities in Nginx, HAproxy, etc. But it would seem like less work to add expressiveness to one of those (via Lua maybe?) than to maintain both Varnish and the separate components needed for both tls ingress and egress.
  - foobarbazetc 5y ago
    Varnish and VCL were the hotness at the time Fastly was coming up so it sort of makes sense, but Varnish also doesn’t support WebSockets, can’t proxy GRPC, etc so they’re very limited in functionality vs CloudFlare. I doubt they’d build it on Varnish today, but it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) and will have to support that for eternity. They can run two or more serving stacks side by side though if it comes to that. And to add to your point, they also have a separate process that speaks QUIC. It’s an interesting tech stack with a lot of technical debt.
    - tyingq 5y ago
      >it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) Ah, okay. I took a look, and it appears they at least didn't allow varnish modules or inline C. But, still, a fairly hefty anchor for the future.
    - 5y ago
jedimastert 5y ago
I assume that the bug isn't 100% fixed yet and the instant they publish how the bug took place 1000 yahoos will immediately try to re-create it.
staticassertion 5y ago
If you find out you have a DOS that can take down the internet you might be wary about sharing details until you've hammered things out.

Fastly doesn't appear to be sharing that detail. Their own blog post is similarly vague about the exact cause. https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage Edit: That blog post does say this: "On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances." The scheduled maintenance on May 12 was this: https://status.fastly.com/incidents/dlsphjqst537 https://status.fastly.com/incidents/dlsphjqst537 Based on that, it sounds like maybe a configuration change could deploy new cache nodes with ip addresses that a customer hasn't explicitly allowed to talk to their backend: "When this change is applied, customers may observe additional origin traffic as new cache nodes retrieve content from origin. Please be sure to check that your origin access lists allow the full range of Fastly IP addresses"

abluecloud 5y ago
Why is everyone banging on about this. It's a blog post from the same day, a decent post mortem takes a while to put together and assuming the bug isn't fully patched across their entire CDN, why would they post the information.
- tyingq 5y ago
  I wasn't "banging on", I was answering why the article didn't mention the cause...because the source didn't either.
- notacoward 5y ago
  Everyone is "banging on" because there are important lessons to be learned from such incidents, and people want to learn. They hunger for more details about the generalizable aspects of the bug, even if a full post mortem that also covers internal processes etc. might take longer to do. Having participated in many post mortems, in many roles, for systems just as complex, I believe it's entirely possible to provide that information the next (not same) day. Is that still setting the bar too high? Perhaps. Fastly deserves kudos for providing even the level of information that they have, since that's already above the pathetic industry standard, but I don't think there's anything bad about wanting more. Defensiveness is the enemy of effective post mortems.
- jffry 5y ago
  Plus it would be weird to present just that specific information, outside of the context of a post mortem / failure chain analysis type discussion.
  - tyingq 5y ago
    That's true, though they are also saying things like "We created a permanent fix for the bug and began deploying it at 17:25.". "Permanent fix" sort of implies they understood the issue really well.
    - jffry 5y ago
      That's my point though. Even though they may understand the immediate flaw in their code that caused the issue, there's not much use (for them or their customers) in just talking in detail about that specific flaw. I'd go so far as to argue that the specifics of the flaw are immaterial right now. At this stage, the important thing is that they have identified a specific code change that was the proximate cause of the issue, and have a mitigation in place. This is contrasted with more mysterious and hard-to-track-down failures. ("We are working to understand why our systems are down and will post another update in 30 minutes") What will take time, and the thing which will be interesting, is failure tree analysis. (You might hear the phrase "failure chain" or "root cause" but IMO it's quite rare for things to be so linear). That can help identify opportunities to improve processes at many different levels of the product lifecycle. Humans are fallible, and there's no way we can write bug-free software, so the solution has to be more robust than "hope that every member of our organization never makes a mistake again"
      - tyingq 5y ago
        Yes, I was saying I would have avoided words like "permanent fix", because it sets unrealistic expectations.
addingnumbers 5y ago
The paragraph you quoted is just describing a side effect of adding nodes. That excerpt appears in every one of their capacity expansion announcements, going back years.
- tyingq 5y ago
  Ah, interesting. Though the change itself reads like it was just adding capacity in one location (Newark). I don't see any other changes mentioned for that date.

Dobbs 5y ago

My uneducated hypothesis is that Fastly runs varnish. Presumably they have some process that collects data from their config system generates the VCL (varnishes psuedo-C configuration language which compiles directly to C). Somehow a customer configured something in such a way that it generated a bad VCL file which then caused it to lose all configuration, or caused one domain to incorrectly garble up traffic for all domains. I can poke plenty of holes in this hypothesis, like fastly likely not deploying configuration to all nodes but only subsets. Looking forward to the deeper post.

tyingq 5y ago
It's interesting to me that they still run Varnish, since it doesn't have https/tls built-in. I do get that VCL is more expressive than similar capabilities in Nginx, HAproxy, etc. But it would seem like less work to add expressiveness to one of those (via Lua maybe?) than to maintain both Varnish and the separate components needed for both tls ingress and egress.
- foobarbazetc 5y ago
  Varnish and VCL were the hotness at the time Fastly was coming up so it sort of makes sense, but Varnish also doesn’t support WebSockets, can’t proxy GRPC, etc so they’re very limited in functionality vs CloudFlare. I doubt they’d build it on Varnish today, but it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) and will have to support that for eternity. They can run two or more serving stacks side by side though if it comes to that. And to add to your point, they also have a separate process that speaks QUIC. It’s an interesting tech stack with a lot of technical debt.
  - tyingq 5y ago
    >it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) Ah, okay. I took a look, and it appears they at least didn't allow varnish modules or inline C. But, still, a fairly hefty anchor for the future.
  - 5y ago

jedimastert 5y ago

I assume that the bug isn't 100% fixed yet and the instant they publish how the bug took place 1000 yahoos will immediately try to re-create it.

staticassertion 5y ago

If you find out you have a DOS that can take down the internet you might be wary about sharing details until you've hammered things out.