4 ms·
Assuming the cloud is 100% hard down and there's nothing you can do to mitigate the primary issue which is not THAT common... Unless you're 100% in a single cl
by AdamJacobMuller 5y ago
Assuming the cloud is 100% hard down and there's nothing you can do to mitigate the primary issue which is not THAT common...
Unless you're 100% in a single cloud in a single region, there's always going to be something which broke which had a dependency on something in the cloud and you can work to mitigate that.
Events like this create increased workload on external and internal customer service teams, people who would normally be engineering can pitch in there. Internally especially hearing from senior dev/ops people helps people.
Shit-post memes about how there's nothing you can do because the cloud is down.
If the outage is on AWS, make up plans for how you should move the infrastructure off AWS onto GCP because AWS has so many outages. If the outage is on GCP, make up plans for how you should move the infrastructure off GCP onto AWS because GCP has so many outages.
Make up plans about how you can scale to multi-cloud muti-region deployments to mitigate against the next cloud outage. Once you finish doing the math realize that you're actually making things more brittle and thus causing more outages and that nobody wants to pay to avoid the hour or three of downtime per year. Realize that the developer who pushed the bad ALTER TABLE statement into production three weeks ago caused a bigger outage than the cloud ever will and begin to question your life choices.
Play some games with the team where you try to guess what sites/services are impacted. Loser has to stay late to close all the tickets generated when cloud comes back up.
Shit-post more memes.