4 ms·

No disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.

• rbanffy 17d ago
  The more dramatic contingency you have to plan for, the more expensive the plan gets. Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.
  • ares623 17d ago
    "Daddy, where were you when the flames reached our house?" "I was in the office, reviewing Terraform plans"
  • jiggawatts 17d ago
    I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities. He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.
  • chasd00 17d ago
    > to-do list becomes securing weapons, vehicles and fuel. I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.
    • mr_mitm 17d ago
      That's probably already true for 7 days or less
      • ehe78qhe 17d ago
        7 days of unreliable electricity wouldn't be unheard of for a very large storm
      • toast0 17d ago
        Large storms regularly result in some customers with lack of utility power for more than 7 days for some customers. When storms take out major transmission lines and roads and bridges, you can end up with some pretty lengthy outages, and fuel deliveries will also be difficult. Look at data center responses from Hurricanes Katrina and Sandy. This guy [1] was onsite for Katrina. Lost utility power on August 29. They did have some access to fuel on Sep 1, but pretty spotty until maybe the 3rd. Looks like power started coming back in some places Sep 8, and maybe widely restored on Sep 14th. I would say, by 7 days in you'll probably have a good idea of if 30 days might not be enough. [1] https://web.archive.org/web/20061206181753/http://interdictor.livejournal.com/2005/08/29/ https://web.archive.org/web/20061206181753/http://interdicto...
        • rbanffy 16d ago
          I need to bring this out: https://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html https://craphound.com/overclocked/Cory_Doctorow_-_Overclocke...
    • SoftTalker 16d ago
      You also need to understand whether the generator backup actually runs everything. Where I work it doesn't. Only "essential" systems get backup power. And if the data center is more than about 5 years old it almost certainly was not planned with adequate backup power to run racks of GPUs.
    • rbanffy 16d ago
      I had a similar discussion in the low 1990s at a large electronic manufacturer that had a pair of Unisys mainframes, one in São Paulo and another in Manaus, in the Amazon region communicating over a satellite link (two large 3m dishes on both places). There was one question of what happens if both fail at the same time, and I pointed that anything that takes out São Paulo and Manaus at the same time is a civilisation ending event, and we shouldn’t worry too much about that. Anyway, we had a load shedding agreement with a large bank across the street from São Paulo and we could switch over to their Unisys mainframe in a matter of minutes, and vice versa.
• rokhayakebe 17d ago
  Did local laws permit?
• deleted 17d ago
  [deleted]
• advisedwang 17d ago
  That depends on the data. If this is EBS or single-AZ S3, then from Amazon's perspective this was correct. Backup responsibly (for any data that does need to be backed up) lives with the customer, and Amazon has no way of knowing about that. EBS data data is unrecoverable, and that's what's reported. Now if this was multi-AZ S3 or whatever then this would be significant. The article does not tell us what products were impacted.
  • altcognito 17d ago
    I was unaware that Amazon even sold single AZ S3. 20% discount. Doesn't seem worth it. By the time I commit to purchasing S3 space, it has to be important data. I get that S3 is convenient and reasonably performant, but it is not cheap at all.
    • kevml 17d ago
      That’s simply not true. I use S3 (well GCS mostly) for data that I wouldn’t be upset if it’s lost. And I pay the zonal discount for it.
      • londons_explore 16d ago
        Google internally has lots of possible redundancy levels for data. They don't sell any of the lower and less reliable levels to the public, I suspect simply because the reputational damage from losing user data is so bad, and the news will take no notice of the fact the user got a discount for less reliable storage.
        • kccqzy 16d ago
          Most of Google's customers wouldn't know how to choose anyways, if these were exposed. My memory was quite hazy but I recall having a discussion with my colleague on choosing which Reed–Solomon code for our Colossus files, and apparently the choice was down to RS(8,3) or RS(9,3). I don't think even as Googlers we really had enough information to make an informed choice. Comparatively it was much easier to decide which cells to use for multi-location replication in Placer.
• cpncrunch 17d ago
  But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.
• yipinwong 17d ago
  Nothing is ever real-time. Eventual consistency leads to some data are not backed up. You talking as if this is some mom-and-pop shop that you run.
• jeremyjh 17d ago
  That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.
• nixass 17d ago
  Offsite to.. where? Sea? Data residency in Gulf states is very strict and basically nothing is leaving the countries
  • firesteelrain 17d ago
    Typically 300 miles geographically but could be hard in some Gulf States
    • XorNot 17d ago
      In a Gulf State 300 miles is still within ballistic missile range and any belligerent is going to target both places if at all. Strictly speaking from a missile defense perspective there's an argument 2 sites are a waste of valuable interceptors.
      • tgsovlerkhgsel 17d ago
        They're likely going to target two datacenters, not the datacenters + your medium sized company's office NAS and the safe in the office manager's home. (Encryption handles confidentiality concerns.)
        • niij 16d ago
          > (Encryption handles confidentiality concerns.) Which is why data residency is such a stupid concept.
          • firesteelrain 16d ago
            Yes and no. For example if you are doing Azure, technically Azure can see tenant traffic I believe and you need to use both a Platform Key and CMK for data rest. VMs need encryption at host turned on too. There is nothing to say that a determined adversary may still get at your data so it needs to stay in country.
            • 16d ago
• sparkling 17d ago
  If a AWS customer chooses to store their data in a single AZ, that is a design choice. AWS is not taking a daily copy of a entire regions S3 cluster and driving it to some warehouse for a "just in case" situation. That is why Multi-AZ exists.
  • bigiain 17d ago
    Isn't S3 claiming eleven nines of data durability? https://aws.amazon.com/s3/storage-classes/ https://aws.amazon.com/s3/storage-classes/ "Additionally, S3 stores data redundantly across a minimum of 3 Availability Zones by default, providing built-in resilience against widespread disaster." I wonder if "can't restore some data" includes any S3 data? I'd expect to lose EC2 instance EBS data in the event of a datacenter being destroyed, but I kinda assume I wouldn't lose S3 data? Now I'm wondering if RDS backups are more like EBS or S3...
    • mitxela 16d ago
      1/f noise strikes again
• toast0 17d ago
  If you had data at two facilities in different countries hundreds of miles apart (about 250 miles between Dubai and Bahrain), that would count as offsite backup most of the time. Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.
• tjwebbnorfolk 17d ago
  Not all data is allowed to leave all countries.
  • lstodd 16d ago
    what a silly notion.
• kjs3 17d ago
  You apparently don't do business out here with the unwashed masses where "whadda mean with all that nonsense? It's cloud...it's by definition safe[1][2]!" is an all too common preconception. [1] That's a quote, including the Boston accent. [2] The only one I had that was better was a C-level who said "why are you asking for all this money for security in Azure. It's Microsoft so it's already secure.". That, too, is a quote.
• deleted 16d ago
  [deleted]
• HannaTao 16d ago
  [dead]