10 ms·

Throwing in my positive hot take among all the negative ones here: the immediate response and blog post from Fastly here is really good. A quick fix, a clear a

Throwing in my positive hot take among all the negative ones here: the immediate response and blog post from Fastly here is really good. A quick fix, a clear apology, enough detail to give an idea of what happened, but not so much detail that there might be a mistake they’ll have to clarify or retract. What more are you looking for? Apart from “not have the bug in the first place” -- and I hope and expect they’ll go into more detail later when they’ve had time for a proper post mortem -- I’d be interested to hear what anyone thinks they could have done better in terms of their immediate firefighting.

oneeyedpigeon 5y ago
I'm sorry, but I disagree. They gave the BBC enough detail that a very misleading headline was produced as a result. True, the main blame lies with the BBC, but it also comes across — to me, anyway, maybe I'm being too cynical — as a bit of an excuse from Fastly.
- mytailorisrich 5y ago
  "But a customer quite legitimately changing their settings had exposed a bug in a software update issued to customers in mid-May, causing "85% of our network to return errors" They are careful to make clear that the customer did nothing wrong and that the problem was a bug in their software.
  - oneeyedpigeon 5y ago
    I know — as I've said, the main blame lies with the bbc. However, as it's reported, it comes across very much as Fastly trying to save face. Maybe the blame is entirely on the bbc, maybe Fastly were naive in thinking that giving them this information wouldn't result in irresponsible headlines.
    - Jenk 5y ago
      How on earth do you figure that is anyone's fault but the BBC? Read Fastly's statement. There is nothing about it blaming the customer(s) at all. There is nothing trying to save face. What is your point here?
      - oneeyedpigeon 5y ago
        > Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors. Is it necessary to refer to "a customer" at all in this statement? What would be problematic if the above were rewritten as something like: > Early June 8, a configuration change triggered a bug in our software, which caused 85% of our network to return errors. The advantage is that you wouldn't get ignorant reporting that "one customer took down the internet". I'm not sure there are disadvantages that net outweigh that.
        laumars 5y ago
        > Is it necessary to refer to "a customer" at all in this statement? That’s how autopsies work. You describe the cause and resolution. The cause was a bug in the customers control panel. They’re not trying to absolve themselves of responsibility.
        Jenk 5y ago
        Yes, because it is explaining that it was a valid *customer* configuration, which is a separate set of concerns from, say, infrastructure config. The important adjective "valid" means it was completely normal/expected input and thus not the fault of the customer. It's perfectly clear you've come at this with a pre-determined agenda of "I bet fastly, like most other public statements after corporate booboos I've seen, will try to shrug this one off as someone else's fault" after reading the BBCs title and haven't bothered to read it at all until now.
        madaxe_again 5y ago
        Not necessarily valid. Could have been a bad entry that passed validation when it shouldn’t have, which would still not be the customer’s fault.
    - jerf 5y ago
      What verbiage exactly are you looking for from Fastly here? I'm hearing "Nobody else did anything wrong, it was 100% a software bug on our end, and we're sorry about that." How much more responsibility are you asking them to take before you would no longer be considering them to "save face"? I'm trying to come up with an ironic exaggeration here, but I can't, because it kinda seems like Fastly has already taken 100% full responsibility and there's no room left for exaggeration.
    - bennyp101 5y ago
      Don't forget that the BBC were also initially affected by this, and jumped on it a lot sooner than most outlets, so they have skin in the game.
      - iso1210 5y ago
        Would love it if it was the BBC that triggered the problem :D
uncertainrhymes 5y ago
If they have a bug that can crash their servers, they likely won't want to publicize the details until it is fully patched. I wouldn't expect that detail for a while.
- alvis 5y ago
  As an ex CTO having lots of fire fighting experience, I wanna give the honor to the team being able to identify such a user triggered bug in such a short period of time. Hardly anyone would anticipate a single user can trigger a meltdown to the internet!
  - artichokeheart 5y ago
    It reminds me of that ancient joke: A QA engineer walks into a bar. He orders a beer. Orders 0 beers. Orders 99999999999 beers. Orders a lizard. Orders -1 beers. Orders a ueicbksjdhd. First real customer walks in and asks where the bathroom is. The bar bursts into flames, killing everyone.
    - dnautics 5y ago
      You forgot the part where the QA engineer ticks off the checklist and gives the bar a pass.
- iainmerrick 5y ago
  The blog post actually says it’s fixed already (but I would definitely expect them to keep the details private until they’re 100% sure, yeah)
thayne 5y ago
I don't blame them for having a bug. I do blame them for having a design that doesn't isolate incident like this (although, it is hard to know how much without more details. And I blame our industry for relying so much on a single company (and that isn't a problem unique to fastly, or even our industry).
- madaxe_again 5y ago
  I would wager that they run a single configuration, as it grants a significant economy of scale, rather than vertical partitioning of their stack, which would require headroom per customer and/or slice. This way you just need global headroom. Having done some similar stuff with varnish in the past (ecommerce platform), they’re likely taking changes in the control panel and deploying them to a global config - and someone put something lethal in that somehow passed validation and got published, and did not parse.
  - aliasEli 5y ago
    This looks a quite likely scenario. But then we still don't know what they fixed, was is the incorrect configuration or the underlying bug? I would expect the former instead of the latter, because it is probably not very difficult or dangerous to change that specific configuration while fixing bugs in the code seems riskier and would probably take more time for testing. We'll see if they will publish a post-mortem. It has become more or less a normal custom these days (and they are frequently quite interesting).
    - dkarp 5y ago
      They were pretty clear about this in their response (linked in the article) Once the immediate effects were mitigated, we turned our attention to fixing the bug and communicating with our customers. We created a permanent fix for the bug and began deploying it at 17:25. So they did both. First reverted the config then later fixed the bug.
- mason55 5y ago
  > And I blame our industry for relying so much on a single company (and that isn't a problem unique to fastly, or even our industry). The problem is that if fastly is the best choice for a company then there's zero incentive for the company to choose another vendor. Everyone acting in their own best interest results in a sub-optimal global outcome. It's actually one of the major problems with the global, winner-takes-all marketplace that's evolving with the internet.

oneeyedpigeon 5y ago

I'm sorry, but I disagree. They gave the BBC enough detail that a very misleading headline was produced as a result. True, the main blame lies with the BBC, but it also comes across — to me, anyway, maybe I'm being too cynical — as a bit of an excuse from Fastly.

mytailorisrich 5y ago
"But a customer quite legitimately changing their settings had exposed a bug in a software update issued to customers in mid-May, causing "85% of our network to return errors" They are careful to make clear that the customer did nothing wrong and that the problem was a bug in their software.
- oneeyedpigeon 5y ago
  I know — as I've said, the main blame lies with the bbc. However, as it's reported, it comes across very much as Fastly trying to save face. Maybe the blame is entirely on the bbc, maybe Fastly were naive in thinking that giving them this information wouldn't result in irresponsible headlines.
  - Jenk 5y ago
    How on earth do you figure that is anyone's fault but the BBC? Read Fastly's statement. There is nothing about it blaming the customer(s) at all. There is nothing trying to save face. What is your point here?
    - oneeyedpigeon 5y ago
      > Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors. Is it necessary to refer to "a customer" at all in this statement? What would be problematic if the above were rewritten as something like: > Early June 8, a configuration change triggered a bug in our software, which caused 85% of our network to return errors. The advantage is that you wouldn't get ignorant reporting that "one customer took down the internet". I'm not sure there are disadvantages that net outweigh that.
      - laumars 5y ago
        > Is it necessary to refer to "a customer" at all in this statement? That’s how autopsies work. You describe the cause and resolution. The cause was a bug in the customers control panel. They’re not trying to absolve themselves of responsibility.
      - Jenk 5y ago
        Yes, because it is explaining that it was a valid *customer* configuration, which is a separate set of concerns from, say, infrastructure config. The important adjective "valid" means it was completely normal/expected input and thus not the fault of the customer. It's perfectly clear you've come at this with a pre-determined agenda of "I bet fastly, like most other public statements after corporate booboos I've seen, will try to shrug this one off as someone else's fault" after reading the BBCs title and haven't bothered to read it at all until now.
        madaxe_again 5y ago
        Not necessarily valid. Could have been a bad entry that passed validation when it shouldn’t have, which would still not be the customer’s fault.
  - jerf 5y ago
    What verbiage exactly are you looking for from Fastly here? I'm hearing "Nobody else did anything wrong, it was 100% a software bug on our end, and we're sorry about that." How much more responsibility are you asking them to take before you would no longer be considering them to "save face"? I'm trying to come up with an ironic exaggeration here, but I can't, because it kinda seems like Fastly has already taken 100% full responsibility and there's no room left for exaggeration.
  - bennyp101 5y ago
    Don't forget that the BBC were also initially affected by this, and jumped on it a lot sooner than most outlets, so they have skin in the game.
    - iso1210 5y ago
      Would love it if it was the BBC that triggered the problem :D

uncertainrhymes 5y ago

If they have a bug that can crash their servers, they likely won't want to publicize the details until it is fully patched. I wouldn't expect that detail for a while.

alvis 5y ago
As an ex CTO having lots of fire fighting experience, I wanna give the honor to the team being able to identify such a user triggered bug in such a short period of time. Hardly anyone would anticipate a single user can trigger a meltdown to the internet!
- artichokeheart 5y ago
  It reminds me of that ancient joke: A QA engineer walks into a bar. He orders a beer. Orders 0 beers. Orders 99999999999 beers. Orders a lizard. Orders -1 beers. Orders a ueicbksjdhd. First real customer walks in and asks where the bathroom is. The bar bursts into flames, killing everyone.
  - dnautics 5y ago
    You forgot the part where the QA engineer ticks off the checklist and gives the bar a pass.
iainmerrick 5y ago
The blog post actually says it’s fixed already (but I would definitely expect them to keep the details private until they’re 100% sure, yeah)

thayne 5y ago

I don't blame them for having a bug. I do blame them for having a design that doesn't isolate incident like this (although, it is hard to know how much without more details. And I blame our industry for relying so much on a single company (and that isn't a problem unique to fastly, or even our industry).

madaxe_again 5y ago
I would wager that they run a single configuration, as it grants a significant economy of scale, rather than vertical partitioning of their stack, which would require headroom per customer and/or slice. This way you just need global headroom. Having done some similar stuff with varnish in the past (ecommerce platform), they’re likely taking changes in the control panel and deploying them to a global config - and someone put something lethal in that somehow passed validation and got published, and did not parse.
- aliasEli 5y ago
  This looks a quite likely scenario. But then we still don't know what they fixed, was is the incorrect configuration or the underlying bug? I would expect the former instead of the latter, because it is probably not very difficult or dangerous to change that specific configuration while fixing bugs in the code seems riskier and would probably take more time for testing. We'll see if they will publish a post-mortem. It has become more or less a normal custom these days (and they are frequently quite interesting).
  - dkarp 5y ago
    They were pretty clear about this in their response (linked in the article) Once the immediate effects were mitigated, we turned our attention to fixing the bug and communicating with our customers. We created a permanent fix for the bug and began deploying it at 17:25. So they did both. First reverted the config then later fixed the bug.
mason55 5y ago
> And I blame our industry for relying so much on a single company (and that isn't a problem unique to fastly, or even our industry). The problem is that if fastly is the best choice for a company then there's zero incentive for the company to choose another vendor. Everyone acting in their own best interest results in a sub-optimal global outcome. It's actually one of the major problems with the global, winner-takes-all marketplace that's evolving with the internet.