8 ms·

I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly

I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly announced in advance) would be a worthwhile value-add. A service that is unable to handle such a failure does not qualify as being ready for deployment. And I'm not just saying that to be a self-righteous pedant, I'm saying it because this kind of failure is statistically likely to happen at some point, so ignoring it is setting yourself up for major (and potentially very expensive) problems if you don't test for it regularly.

kall 5y ago
At the project level maybe. At the AWS level, that‘s ridiculous. A lot of services can live with one major outage a year, but not once a month. Completely failing when their region goes down is reasonable.
- deleted 5y ago
  [deleted]
- peterkelly 5y ago
  Region failures can be non-recoverable (e.g. natural disaster). One major outage per year doesn't necessarily mean just a few hours of downtime, it could mean having to redeploy your entire service somewhere else, which could take several days or more if you haven't prepared for it. How many of those services can live with that?
  - stdbrouw 5y ago
    > How many of those services can live with that? Apparently, many can! Look at how many companies choose not to pay ransom when hit with a ransomware attack, or prefer to negotiate for days instead of buckling straight away, even if it means operations are crippled for weeks. They don't typically go bankrupt afterwards, everyone coped, life moves on.
ec109685 5y ago
Netflix has a chaos gorilla that does something like that: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html And even more now: http://techblog.netflix.com/2013/12/active-active-for-multi-regional.html http://techblog.netflix.com/2013/12/active-active-for-multi-...
- pojzon 5y ago
  Chaos Engineering was more or less created for a service mesh of microservices. The companies we talk here about are most likely running a monolith app on Centos6..
  - blantonl 5y ago
    * raises hand :)
- smartbit 5y ago
  Therefor I didn’t expect Netflix to go down. Would be interesting to learn in a postmortem why Netflix had issues despite Chaos Gorilla.
  - ec109685 5y ago
    It seems like the fact that Route53 went down for changes was a big part of it.
- bobsmooth 5y ago
  I wonder how the simian army has evolved in the past decade.
stdbrouw 5y ago
Smallish non-tech companies typically have single points of failure that are much more severe than a server/service being down for a couple of hours. Accountant on holiday? Guess we're not writing any invoices this week. Two of our drivers are ill? Guess we'll have to let all of this stuff pile up here for a bit. By comparison, a public-facing service or intranet being down for a few hours hardly merits attention.
sciurus 5y ago
Shutting down availability zones I could almost see, since it's not that much more difficult or expensive to architect to handle that gracefully. But an entire region? I've never worked anywhere that decided being multi-region was a good tradeoff. At best we've replicated data to another region and had some of our management services there, so in the absolute worst case (which would need to be much worse than yesterday) we could rebuild our product's infrastructure there. Do I agree with this approach? Not in all possible cases of course, but for my employers? Overall, yes. It mitigates the highest impact risks. Going further would have significant complexity and costs. Those companies success or failure haven't been impacted by their multi-region strategy AFAICT.
unionpivo 5y ago
Most services and applications can occasionally have unplanned downtime. Many have also planed downtimes sometimes measured in half a day or more. Services that truly need 100% uptime (and by that I don't mean what management say they WANT, but they are prepared to PAY) are a tiny minority, to the point that I imagine most software developers never work on one. Even setups that i have seen to handle such cases, usually had some single point of failure somewhere. And lest face it. Even if you do multi region on AWS, that won't protect you next time someone screws up BGP/switches/DNS or whatever and every region goes down for a while. In that case you better have failover to some other cloud vendor. Even planes or other safety critical equipment is not 100% failure proof.

kall 5y ago

At the project level maybe. At the AWS level, that‘s ridiculous. A lot of services can live with one major outage a year, but not once a month. Completely failing when their region goes down is reasonable.

deleted 5y ago
[deleted]
peterkelly 5y ago
Region failures can be non-recoverable (e.g. natural disaster). One major outage per year doesn't necessarily mean just a few hours of downtime, it could mean having to redeploy your entire service somewhere else, which could take several days or more if you haven't prepared for it. How many of those services can live with that?
- stdbrouw 5y ago
  > How many of those services can live with that? Apparently, many can! Look at how many companies choose not to pay ransom when hit with a ransomware attack, or prefer to negotiate for days instead of buckling straight away, even if it means operations are crippled for weeks. They don't typically go bankrupt afterwards, everyone coped, life moves on.

ec109685 5y ago

Netflix has a chaos gorilla that does something like that: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html And even more now: http://techblog.netflix.com/2013/12/active-active-for-multi-regional.html http://techblog.netflix.com/2013/12/active-active-for-multi-...

pojzon 5y ago
Chaos Engineering was more or less created for a service mesh of microservices. The companies we talk here about are most likely running a monolith app on Centos6..
- blantonl 5y ago
  * raises hand :)
smartbit 5y ago
Therefor I didn’t expect Netflix to go down. Would be interesting to learn in a postmortem why Netflix had issues despite Chaos Gorilla.
- ec109685 5y ago
  It seems like the fact that Route53 went down for changes was a big part of it.
bobsmooth 5y ago
I wonder how the simian army has evolved in the past decade.

stdbrouw 5y ago

Smallish non-tech companies typically have single points of failure that are much more severe than a server/service being down for a couple of hours. Accountant on holiday? Guess we're not writing any invoices this week. Two of our drivers are ill? Guess we'll have to let all of this stuff pile up here for a bit. By comparison, a public-facing service or intranet being down for a few hours hardly merits attention.

sciurus 5y ago

Shutting down availability zones I could almost see, since it's not that much more difficult or expensive to architect to handle that gracefully. But an entire region? I've never worked anywhere that decided being multi-region was a good tradeoff. At best we've replicated data to another region and had some of our management services there, so in the absolute worst case (which would need to be much worse than yesterday) we could rebuild our product's infrastructure there. Do I agree with this approach? Not in all possible cases of course, but for my employers? Overall, yes. It mitigates the highest impact risks. Going further would have significant complexity and costs. Those companies success or failure haven't been impacted by their multi-region strategy AFAICT.

unionpivo 5y ago

Most services and applications can occasionally have unplanned downtime. Many have also planed downtimes sometimes measured in half a day or more. Services that truly need 100% uptime (and by that I don't mean what management say they WANT, but they are prepared to PAY) are a tiny minority, to the point that I imagine most software developers never work on one. Even setups that i have seen to handle such cases, usually had some single point of failure somewhere. And lest face it. Even if you do multi region on AWS, that won't protect you next time someone screws up BGP/switches/DNS or whatever and every region goes down for a while. In that case you better have failover to some other cloud vendor. Even planes or other safety critical equipment is not 100% failure proof.