4 ms·

"Fastly senior engineering executive Nick Rockwell said: "This outage was broad and severe - and we're truly sorry for the impact to our customers and everyone

"Fastly senior engineering executive Nick Rockwell said: "This outage was broad and severe - and we're truly sorry for the impact to our customers and everyone who relies on them."" I wonder here, are they sorry enough to run the company on two different tech and software stacks and data centers? Like people don't buy disk drives (SSDs) not all from the same vendor? How much would they spend for the "sorry".

mhandley 5y ago
You'd have to consider what actually happens when 50% of your infrastructure goes down. Can the remaining 50% cope with 100% of the load? If not, then you still get a complete failure. So then the question becomes can you reprovision between stack A and stack B very rapidly, both running from the same hardware pool, while 50% of your infrastructure is down. Now you've introduced the potential for correlated failures (single hardware pool and network), plus added complexity and load due to reprovisioning just when things are already overloaded. Not easy to get this right, so might not actually increase reliability, as now you've got two different stacks that can each independently fail and get you into this mess.
ceejayoz 5y ago
Without more details on what/where the "software bug" was, it's hard to say if that would've helped at all.
stingraycharles 5y ago
That only protects against one type of failure, eg one of your vendors going down. The article suggests this was a problem in their own software, which in all likelihood would not be protected against by using different stacks.
- KingOfCoders 5y ago
  Two developers and two product managers would not create the same bugs but different ones I'd think. From writing thousands of correct lines and implementing features without bugs, it would be a coincidende (except perhaps not well understood complex parts) where two developers create the same bugs. Reading studies about bugs in the past where developers where told to write some code resulted in different bugs.
  - stingraycharles 5y ago
    Are there any organizations out there that actually make multiple versions of their product and deploy it together, in real time, all the while being completely agnostic to the end user? It seems like it will be very hard to justify the immense costs for this.
dap 5y ago
I’ve heard this before but I’m skeptical about this approach. It’s got to cost close to 2x to develop and maintain two stacks — if they’re sharing a lot, that would defeat the point. Then you’d have two stacks tested about as well as the one today. Instead, you could put that investment towards testing and improving quality on the one stack. I know a second stack hopefully has different bugs. But is that likely (to a meaningful extent)? Reimplementations often reintroduce old issues (which suggests many people make similar mistakes), plus it’s hard to imagine the first stack not influencing the second in various ways.
tibbetts 5y ago
Redundant implementations only help with uncorrelated problems. With complex software, problems are often quite correlated across multiple implementations. Add to that the additional complexity of managing those multiple implementations and the potential for problems may be net worse. The consensus among safety experts is that multiple implementations are a bad approach to safety or reliability.
- KingOfCoders 5y ago
  I would assume different developers and product managers create different bugs.
spicybright 5y ago
Making your infra heterogeneous is very underrated. Or at least I don't hear of many companies doing that, even large ones. How do you go about different software though, have one center running a version or two behind to fallover to?
- aliasEli 5y ago
  In general it is expensive. In most cases you will need experts that need knowledge of both system A and system B. Those people are more difficult to find. Also you have a higher chance of errors due to confusion between the two systems.

mhandley 5y ago

You'd have to consider what actually happens when 50% of your infrastructure goes down. Can the remaining 50% cope with 100% of the load? If not, then you still get a complete failure. So then the question becomes can you reprovision between stack A and stack B very rapidly, both running from the same hardware pool, while 50% of your infrastructure is down. Now you've introduced the potential for correlated failures (single hardware pool and network), plus added complexity and load due to reprovisioning just when things are already overloaded. Not easy to get this right, so might not actually increase reliability, as now you've got two different stacks that can each independently fail and get you into this mess.

ceejayoz 5y ago

Without more details on what/where the "software bug" was, it's hard to say if that would've helped at all.

stingraycharles 5y ago

That only protects against one type of failure, eg one of your vendors going down. The article suggests this was a problem in their own software, which in all likelihood would not be protected against by using different stacks.

KingOfCoders 5y ago
Two developers and two product managers would not create the same bugs but different ones I'd think. From writing thousands of correct lines and implementing features without bugs, it would be a coincidende (except perhaps not well understood complex parts) where two developers create the same bugs. Reading studies about bugs in the past where developers where told to write some code resulted in different bugs.
- stingraycharles 5y ago
  Are there any organizations out there that actually make multiple versions of their product and deploy it together, in real time, all the while being completely agnostic to the end user? It seems like it will be very hard to justify the immense costs for this.

dap 5y ago

I’ve heard this before but I’m skeptical about this approach. It’s got to cost close to 2x to develop and maintain two stacks — if they’re sharing a lot, that would defeat the point. Then you’d have two stacks tested about as well as the one today. Instead, you could put that investment towards testing and improving quality on the one stack. I know a second stack hopefully has different bugs. But is that likely (to a meaningful extent)? Reimplementations often reintroduce old issues (which suggests many people make similar mistakes), plus it’s hard to imagine the first stack not influencing the second in various ways.

tibbetts 5y ago

Redundant implementations only help with uncorrelated problems. With complex software, problems are often quite correlated across multiple implementations. Add to that the additional complexity of managing those multiple implementations and the potential for problems may be net worse. The consensus among safety experts is that multiple implementations are a bad approach to safety or reliability.

KingOfCoders 5y ago
I would assume different developers and product managers create different bugs.

spicybright 5y ago

Making your infra heterogeneous is very underrated. Or at least I don't hear of many companies doing that, even large ones. How do you go about different software though, have one center running a version or two behind to fallover to?

aliasEli 5y ago
In general it is expensive. In most cases you will need experts that need knowledge of both system A and system B. Those people are more difficult to find. Also you have a higher chance of errors due to confusion between the two systems.