3 ms·

Interesting to see it's been a loss of power that caused this. Usually the better datacenters have multiple levels of power redundancy including emergency backu

Interesting to see it's been a loss of power that caused this. Usually the better datacenters have multiple levels of power redundancy including emergency backup generators.

throwawaymaths 4y ago
Insert clip of O'Brien explaining to cardassians why there are backups for backups
- laumars 4y ago
  In case anyone is unaware of the reference, that’s taken from Star Trek Deep Space 9 https://youtu.be/UaPkSU8DNfY https://youtu.be/UaPkSU8DNfY
jmartens 4y ago
Ya, am I surprised by this too. Like, you have one job, keep the power on.
packetslave 4y ago
It depends entirely on how AWS architected their power redundancy. Given that the outage affected a portion of one DC in one AZ, we can make some assumptions, but the truth is we just don't know. It could be that their shared-fate scope is an entire data hall, or a set of rows, or even an entire building given that an AZ is made up of multiple datacenters. I don't know that AWS has ever published any kind of sub-AZ guarantees around reliability. Datacenter power has all kinds of interesting failure modes. I've seen outages caused by a cat climbing into a substation, rats building a nest in a generator, fire-fighting in another part of the building causing flooding in the high-voltage switching room, etc.
- latchkey 4y ago
  Our best was a bird landing on a transformer up on a pole. Installed a fake Eagle after that.
  - mhb 4y ago
    Given the scope of the effort invested in attempting to prevent duck and goose crap on the world's docks, I'm skeptical that this tactic is effective.
    - latchkey 4y ago
      Shrug... the datacenter is land locked (different animal species) and the problem hasn't happened again in multiple years. I think you're taking the Eagle a bit too seriously though... if we didn't do anything how would we know? It isn't like this was an expensive thing to try out.
      - mhb 4y ago
        OK. It's just that I am one of those people who have tried to solve the duck/goose problem and would be delighted if a fake eagle or owl worked.

Interesting to see it's been a loss of power that caused this. Usually the better datacenters have multiple levels of power redundancy including emergency backup generators.

throwawaymaths 4y ago

Insert clip of O'Brien explaining to cardassians why there are backups for backups

laumars 4y ago
In case anyone is unaware of the reference, that’s taken from Star Trek Deep Space 9 https://youtu.be/UaPkSU8DNfY https://youtu.be/UaPkSU8DNfY

jmartens 4y ago

Ya, am I surprised by this too. Like, you have one job, keep the power on.

packetslave 4y ago

It depends entirely on how AWS architected their power redundancy. Given that the outage affected a portion of one DC in one AZ, we can make some assumptions, but the truth is we just don't know. It could be that their shared-fate scope is an entire data hall, or a set of rows, or even an entire building given that an AZ is made up of multiple datacenters. I don't know that AWS has ever published any kind of sub-AZ guarantees around reliability. Datacenter power has all kinds of interesting failure modes. I've seen outages caused by a cat climbing into a substation, rats building a nest in a generator, fire-fighting in another part of the building causing flooding in the high-voltage switching room, etc.

latchkey 4y ago
Our best was a bird landing on a transformer up on a pole. Installed a fake Eagle after that.
- mhb 4y ago
  Given the scope of the effort invested in attempting to prevent duck and goose crap on the world's docks, I'm skeptical that this tactic is effective.
  - latchkey 4y ago
    Shrug... the datacenter is land locked (different animal species) and the problem hasn't happened again in multiple years. I think you're taking the Eagle a bit too seriously though... if we didn't do anything how would we know? It isn't like this was an expensive thing to try out.
    - mhb 4y ago
      OK. It's just that I am one of those people who have tried to solve the duck/goose problem and would be delighted if a fake eagle or owl worked.