4 ms·
I've been working on that for a client since yesterday (as a fractional CTO). Pretty hectic, basically nothing really works and we don't know yet if all data is
by RamblingCTO 7mo ago
I've been working on that for a client since yesterday (as a fractional CTO). Pretty hectic, basically nothing really works and we don't know yet if all data is lost or if anything is recoverable or when AWS UAE will become functional again so we can recover that region.
Finally, I have a very good argument for multi-region deployments ;))
that's my go to website atm: https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status
- chinathrow 7mo agoAny backups?
- TacticalCoder 7mo ago> Any backups? Yup something has to be said too about good old offline backups on fat tapes.
- crossroadsguy 7mo agoSeverity: Disrupted So if data won't be recoverable you all will mark it something like "Status: FUBAR" or some equivalent term?
- Hamuko 7mo agoWe didn't do multi-region deployments, but we did store database backups in a separate region just in case something really bad happened and our AWS region became unavailable. Also had a plan/some ready Terraform stuff in order to start setting up a deployment if it became apparent that the region wasn't coming back anytime soon. IMO, if you're using AWS and not replicating your data somewhere else, this should be an eye-opener for you.
- RamblingCTO 7mo agoNot sure why everyone read this as me doing anything here, I'm a fractional CTO, which is kind of an advisor. Nothing invaluable will be lost tho. It's not the core platform, just a localized version for specific customers in the region.
- richsouth 7mo agoWhat do you mean 'finally' - surely 'redundancy' or 'natural disaster' is reason enough.
- doubled112 7mo agoI don't think this changes anything. It is always the same argument for me. "How often do those happen?"
- RamblingCTO 7mo agoThat was my point: now we have a very specific case we can argue. I always used the "what if a plane crashes into the data center" argument. Now I can say: in one of my engagements, we lost a datacenter completely because of a drone attack. That's a first for AWS as well, but we can draw from reality, not hypotheticals.
- RamblingCTO 7mo agoTo a lot of managers/startup execs this is something like "we won't ever need this". And I'd agree to some extent for not so important/rebuildable services. Depends on what you need. In startups, you don't have infinite time you can spend on stuff. But this makes a good case: if a geographical region only has one AWS region, don't keep data or run services that can't be easily rebuild somewhere else. In europe you can just pick two AWS regions and you stay in the same regions, UAE not so much.
- tw04 7mo agoThere’s a reason literally every enterprise of any size in the early 00s not only had a DR site, but a DR plan they would test multiple times per year. 1. Just having a copy in two places doesn’t mean the copies are both good or that you can get DR online in any reasonable amount of time. 2. You quickly figure out all the things you thought “weren’t important” that prevent you from actually doing a successful DR test. They amount of things that “cloud first” people just assume naturally takes care of itself because it’s in the cloud always amuses me.
- SirFatty 7mo agoYou don't already have a DR plan in place?
- RamblingCTO 7mo agoOf course, no worries. Nothing irreplaceable will be lost ;) It was meant as a general example to be used in future arguments for everybody in tech leadership.
- jamesfinlayson 7mo agoSo AWS's availability zone architecture hasn't helped in this case?
- MagicMoonlight 7mo agoIf a whole region is bombed then you wouldn’t expect that region to stay up.
- RamblingCTO 7mo agoAll AZs were affected