6 ms·

The section about calculating the availability of hard and redundant dependencies ignores the fact that systems often fail in tandem. For e.g. you might have a

The section about calculating the availability of hard and redundant dependencies ignores the fact that systems often fail in tandem. For e.g. you might have a primary and secondary database in different AZs, each with 99.9% availability. This gives read operations a hypothetical uptime of 6-9s. But hidden SPOFs like operator error, VM infra failing, load balancer or other networking outages can make the 0.01% failure time overlap, blowing away your 6-9s dependency guarantee.

Tinned_Tuna 7y ago
p.14: "Because 99.999% availability provides for less than 5 minutes of downtime per year, every operation performed to the system in production will need to be automated and tested with the same level of care. With a 5 minute per year budget, human judgement and action is completely off the table for failure recovery"
- candiodari 7y ago
  That just gets you back to automated systems, for example, causing their own failure, then responding to it by causing more failure. For example, a relatively common occurence is a BGP link getting saturated. Get that situation bad enough and the BGP session will go down, which will redirect all that traffic to another link with another BGP session, which then proceeds to go down. Meanwhile the original session comes back up and ... And then the failures synchronize and cause a third link to go down (each time taking more traffic with it and therefore causing failures faster). The second issue discussed is that the math in the statistics only works if the failures never synchronize. That's true for a lot of statistical analyses and mostly people ... just don't care. Yes that makes those analyses wrong. But we don't have a better way of doing those analyses.
crankylinuxuser 7y ago
(erased)
- FigmentEngine 7y ago
  This is not true. AWS provides multiple constucts to help isolate failures, such as Availability Zones that provide seperate locations, and Regions that are designed to be completely isolated from each other. You can also use different AWS Accounts to provide logical seperation of your different workloads and limit management.
aarghh 7y ago
Couldn't agree with you more. Specifically, certain (poorly-thought out) failure handling mechanisms actually end up transmitting the very condition that triggered the first failure to other parts of the system - thus creating cascading failures.
koolba 7y ago
Many run into this when they learn the hard way that “RAID is not a backup” as wiping out all your data on one set of disks also wiped it on the other. Ditto for naive replication with no rollback capability. It’s a fancier version of the same thing. “DROP TABLE customer” “Oh crap! Better failover to DR. Oh crap! It’s dropped there too?!”
- earless1 7y ago
  exponentially delayed replication down the replica chain...
  - javajosh 7y ago
    Why increase the interval at all? Why not just equally, or at most linearly increasing, delayed replication?
    - nine_k 7y ago
      With replication at 1 minute intervals and ten replicas, you can step back at most 10 minutes. Same with replicas at 1, 2, 4, 8, etc minutes, ten replicas let you step back up to 17 hours. Much more time to detect and correct a catastrophic error.
juskrey 7y ago
Out of the model risks is what usually kills you.
- quirmian 7y ago
  This is a tautology, because if the risk was successfully modelled, it would not be able to kill you.
  - alextheparrot 7y ago
    What’s incorrect about accepting a risk that could kill you, such as flying on air planes or becoming a soldier? Another example is going all-in during a poker game, you’re simply saying you’re ok with that severe risk for the corresponding upsides.
    - quirmian 7y ago
      My point is that stating “out of model risks is what kills you” is a tautology. For systems where known risks are modelled, it’s highly likely that any “fatality” or equivalent events are controlled for. If you were to then look at the “fatalities” that occur after that, I would expect them to arise due to risks not currently known or modelled in the system. I make no comment around acceptance of risk, “fatal” or otherwise. :)
  - juskrey 7y ago
    Risk not included in your model is still a risk
  - vinceguidry 7y ago
    Not necessarily, a risk can be part of the model but not accurately quantified. Having a model doesn't mean it's perfect.

Tinned_Tuna 7y ago

p.14: "Because 99.999% availability provides for less than 5 minutes of downtime per year, every operation performed to the system in production will need to be automated and tested with the same level of care. With a 5 minute per year budget, human judgement and action is completely off the table for failure recovery"

candiodari 7y ago
That just gets you back to automated systems, for example, causing their own failure, then responding to it by causing more failure. For example, a relatively common occurence is a BGP link getting saturated. Get that situation bad enough and the BGP session will go down, which will redirect all that traffic to another link with another BGP session, which then proceeds to go down. Meanwhile the original session comes back up and ... And then the failures synchronize and cause a third link to go down (each time taking more traffic with it and therefore causing failures faster). The second issue discussed is that the math in the statistics only works if the failures never synchronize. That's true for a lot of statistical analyses and mostly people ... just don't care. Yes that makes those analyses wrong. But we don't have a better way of doing those analyses.

crankylinuxuser 7y ago

(erased)

FigmentEngine 7y ago
This is not true. AWS provides multiple constucts to help isolate failures, such as Availability Zones that provide seperate locations, and Regions that are designed to be completely isolated from each other. You can also use different AWS Accounts to provide logical seperation of your different workloads and limit management.

aarghh 7y ago

Couldn't agree with you more. Specifically, certain (poorly-thought out) failure handling mechanisms actually end up transmitting the very condition that triggered the first failure to other parts of the system - thus creating cascading failures.

koolba 7y ago

Many run into this when they learn the hard way that “RAID is not a backup” as wiping out all your data on one set of disks also wiped it on the other. Ditto for naive replication with no rollback capability. It’s a fancier version of the same thing. “DROP TABLE customer” “Oh crap! Better failover to DR. Oh crap! It’s dropped there too?!”

earless1 7y ago
exponentially delayed replication down the replica chain...
- javajosh 7y ago
  Why increase the interval at all? Why not just equally, or at most linearly increasing, delayed replication?
  - nine_k 7y ago
    With replication at 1 minute intervals and ten replicas, you can step back at most 10 minutes. Same with replicas at 1, 2, 4, 8, etc minutes, ten replicas let you step back up to 17 hours. Much more time to detect and correct a catastrophic error.

juskrey 7y ago

Out of the model risks is what usually kills you.

quirmian 7y ago
This is a tautology, because if the risk was successfully modelled, it would not be able to kill you.
- alextheparrot 7y ago
  What’s incorrect about accepting a risk that could kill you, such as flying on air planes or becoming a soldier? Another example is going all-in during a poker game, you’re simply saying you’re ok with that severe risk for the corresponding upsides.
  - quirmian 7y ago
    My point is that stating “out of model risks is what kills you” is a tautology. For systems where known risks are modelled, it’s highly likely that any “fatality” or equivalent events are controlled for. If you were to then look at the “fatalities” that occur after that, I would expect them to arise due to risks not currently known or modelled in the system. I make no comment around acceptance of risk, “fatal” or otherwise. :)
- juskrey 7y ago
  Risk not included in your model is still a risk
- vinceguidry 7y ago
  Not necessarily, a risk can be part of the model but not accurately quantified. Having a model doesn't mean it's perfect.