7 ms·

I don't blame them for having a bug. I do blame them for having a design that doesn't isolate incident like this (although, it is hard to know how much without

by thayne 5y ago

I don't blame them for having a bug. I do blame them for having a design that doesn't isolate incident like this (although, it is hard to know how much without more details. And I blame our industry for relying so much on a single company (and that isn't a problem unique to fastly, or even our industry).

madaxe_again 5y ago
I would wager that they run a single configuration, as it grants a significant economy of scale, rather than vertical partitioning of their stack, which would require headroom per customer and/or slice. This way you just need global headroom. Having done some similar stuff with varnish in the past (ecommerce platform), they’re likely taking changes in the control panel and deploying them to a global config - and someone put something lethal in that somehow passed validation and got published, and did not parse.
- aliasEli 5y ago
  This looks a quite likely scenario. But then we still don't know what they fixed, was is the incorrect configuration or the underlying bug? I would expect the former instead of the latter, because it is probably not very difficult or dangerous to change that specific configuration while fixing bugs in the code seems riskier and would probably take more time for testing. We'll see if they will publish a post-mortem. It has become more or less a normal custom these days (and they are frequently quite interesting).
  - dkarp 5y ago
    They were pretty clear about this in their response (linked in the article) Once the immediate effects were mitigated, we turned our attention to fixing the bug and communicating with our customers. We created a permanent fix for the bug and began deploying it at 17:25. So they did both. First reverted the config then later fixed the bug.
mason55 5y ago
> And I blame our industry for relying so much on a single company (and that isn't a problem unique to fastly, or even our industry). The problem is that if fastly is the best choice for a company then there's zero incentive for the company to choose another vendor. Everyone acting in their own best interest results in a sub-optimal global outcome. It's actually one of the major problems with the global, winner-takes-all marketplace that's evolving with the internet.
- yomly 5y ago
  Do you roll your own power grid? Do you roll your own ISP + telecoms network? As a software engineer I live by the ethos that coupling and dependency is bad, but if you unravel the layers you start to realise much of our life is centralised: Roads, trains, water, electricity, internet These are quite consolidated and any of these going down would be very disruptive to our lives. Connected software, ie the internet, is still quite new. Being charitable, are these just growing pains in the journey to building out foundational infrastructure?
  - iso1210 5y ago
    My datacentres have two sources of power (plus internal UPS), two main internet lines to two different exchange points (and half a dozen others), plenty of bottled water. At home I have emergency power, water and internet. If the trains stop I drive, if the car breaks I take the train.
    - 411111111111111 5y ago
      But having everything redundantly available costs money. While some redundancy is easy to justify... At some point it becomes hard when the MBA wants to cut costs so he gets a bigger bonus. There is even a competitive advantage in living with the risk, as you have less costs and overhead... Sure, you might have an outage once every x years for a few minutes... But that's obviously the fault of the development team, duh
      - 5y ago
iseanstevens 5y ago
Sometimes, it’s something as small as a missing parentheses that makes things go wrong. (Slight joke here)

by thayne 5y ago

madaxe_again 5y ago

I would wager that they run a single configuration, as it grants a significant economy of scale, rather than vertical partitioning of their stack, which would require headroom per customer and/or slice. This way you just need global headroom. Having done some similar stuff with varnish in the past (ecommerce platform), they’re likely taking changes in the control panel and deploying them to a global config - and someone put something lethal in that somehow passed validation and got published, and did not parse.

aliasEli 5y ago
This looks a quite likely scenario. But then we still don't know what they fixed, was is the incorrect configuration or the underlying bug? I would expect the former instead of the latter, because it is probably not very difficult or dangerous to change that specific configuration while fixing bugs in the code seems riskier and would probably take more time for testing. We'll see if they will publish a post-mortem. It has become more or less a normal custom these days (and they are frequently quite interesting).
- dkarp 5y ago
  They were pretty clear about this in their response (linked in the article) Once the immediate effects were mitigated, we turned our attention to fixing the bug and communicating with our customers. We created a permanent fix for the bug and began deploying it at 17:25. So they did both. First reverted the config then later fixed the bug.

mason55 5y ago

> And I blame our industry for relying so much on a single company (and that isn't a problem unique to fastly, or even our industry). The problem is that if fastly is the best choice for a company then there's zero incentive for the company to choose another vendor. Everyone acting in their own best interest results in a sub-optimal global outcome. It's actually one of the major problems with the global, winner-takes-all marketplace that's evolving with the internet.

yomly 5y ago
Do you roll your own power grid? Do you roll your own ISP + telecoms network? As a software engineer I live by the ethos that coupling and dependency is bad, but if you unravel the layers you start to realise much of our life is centralised: Roads, trains, water, electricity, internet These are quite consolidated and any of these going down would be very disruptive to our lives. Connected software, ie the internet, is still quite new. Being charitable, are these just growing pains in the journey to building out foundational infrastructure?
- iso1210 5y ago
  My datacentres have two sources of power (plus internal UPS), two main internet lines to two different exchange points (and half a dozen others), plenty of bottled water. At home I have emergency power, water and internet. If the trains stop I drive, if the car breaks I take the train.
  - 411111111111111 5y ago
    But having everything redundantly available costs money. While some redundancy is easy to justify... At some point it becomes hard when the MBA wants to cut costs so he gets a bigger bonus. There is even a competitive advantage in living with the risk, as you have less costs and overhead... Sure, you might have an outage once every x years for a few minutes... But that's obviously the fault of the development team, duh
    - 5y ago

iseanstevens 5y ago

Sometimes, it’s something as small as a missing parentheses that makes things go wrong. (Slight joke here)