4 ms·
> No, they say to use AZ's Using 3 AZs in us-east-1 won't save you. I guess a demanding customer would have said 'you should have implemented disaster recover
by robrtsql 3y ago
> No, they say to use AZ's
Using 3 AZs in us-east-1 won't save you.
I guess a demanding customer would have said 'you should have implemented disaster recovery so you could failover to us-east-2' but that's easier said than done. The more regional AWS services you adopt, the bigger the impact is. How does one recover from a regional outage if their pipeline is in that region?
- fnordpiglet 3y agoI’ve worked for several systemically important megacorps where certain things had to not only run cross region but also cross provider. It’s absurdly difficult, and only should be done if you need five or more 9’s of availability. Almost nothing actually does.
- fbellag 3y agoWhat I did once I was in the position of _having_ to provide that level of support, was to run the pipeline in a third region, different from the "prod" ones. That way, worst case you can't do deployments during the outage... Another alternative studied was to use a thirdparty ci/cd service, outside of our network. It was discarded bc you never know where that would actually run
- robrtsql 3y ago> It was discarded bc you never know where that would actually run Yep, I considered that switching to GitHub Actions would _theoretically_ eliminate the need for disaster recovery for CI/CD (since the handling of disasters is out of your hands) but in practice their SLA is far worse than just running CodePipeline in a single region.
- fbellag 3y agoYeah, that's why we went with a third region instead. But, at the end of the day, if _only_ changes are affected for a couple of hours, that wouldn't impact the service that much