5 ms·

When you're working at this scale, geographic redundancy of DCs is commonplace. I know of several cases where entire DCs were destroyed, and production continu

by Hoff 16y ago

When you're working at this scale, geographic redundancy of DCs is commonplace. I know of several cases where entire DCs were destroyed, and production continued unabated. There are commercial platforms that are explicitly designed for this sort of DC redundancy over 800+ km spans. Where you have 400 km between various volumes in your RAIDset.

mkr-hn 16y ago
Do you have links? That's a scenario I've always been curious about, and would like to read about situations where it actually happened.
- Hoff 16y ago
  http://h71000.www7.hp.com/openvms/brochures/commerzbank/commerzbank.pdf http://h71000.www7.hp.com/openvms/brochures/commerzbank/comm... A moderately technical article on some of the issues that arise, and (for your question) see the end of the following: http://h71000.www7.hp.com/openvms/journal/v1/disastertol.pdf http://h71000.www7.hp.com/openvms/journal/v1/disastertol.pdf
  - wallflower 16y ago
    Fascinating. "At 2 a.m. on Dec. 29, 1999, an active stock market trading day, the audio alert on a UPS system alarmed a security guard on his first day on the job. He pressed the emergency power-off switch, taking down the entire datacenter. Because a disaster-tolerant OpenVMS cluster was in place, the cluster continued to run at the opposite site, with no disruption. The brokerage ran through that stock-trading day on the one site alone, and performed a shadow full-copy operation to restore redundancy in the evening, after trading hours. They procured a replacement for the failed security guard by the next day." The last sentence.
    - mkr-hn 16y ago
      It's a little disturbing that a security guard had the ability to shut the whole thing down. Is that normal? I would think they'd keep the power systems under lock and key, or at least under the control of someone who knew what to do when a UPS starts making noise.
      - jauer 16y ago
        Nope. Data centers are really easy to shut off. Emergency Power Off buttons are often required by code to shut down breakers and UPS. I've heard that this is to prevent loss of life if a tech grabs a livewire and/or let the fire department know that power is removed before they start spraying water.
      - throw_away 16y ago
        The big red buttons are often there as a safety measure in case someone starts getting electrocuted. At very least, though, these very tempting buttons should be labeled in very dire language that they're not to be pressed unless either a) someone is about to die or b) you actually know what you're doing.

by Hoff 16y ago

mkr-hn 16y ago

Do you have links? That's a scenario I've always been curious about, and would like to read about situations where it actually happened.

Hoff 16y ago
http://h71000.www7.hp.com/openvms/brochures/commerzbank/commerzbank.pdf http://h71000.www7.hp.com/openvms/brochures/commerzbank/comm... A moderately technical article on some of the issues that arise, and (for your question) see the end of the following: http://h71000.www7.hp.com/openvms/journal/v1/disastertol.pdf http://h71000.www7.hp.com/openvms/journal/v1/disastertol.pdf
- wallflower 16y ago
  Fascinating. "At 2 a.m. on Dec. 29, 1999, an active stock market trading day, the audio alert on a UPS system alarmed a security guard on his first day on the job. He pressed the emergency power-off switch, taking down the entire datacenter. Because a disaster-tolerant OpenVMS cluster was in place, the cluster continued to run at the opposite site, with no disruption. The brokerage ran through that stock-trading day on the one site alone, and performed a shadow full-copy operation to restore redundancy in the evening, after trading hours. They procured a replacement for the failed security guard by the next day." The last sentence.
  - mkr-hn 16y ago
    It's a little disturbing that a security guard had the ability to shut the whole thing down. Is that normal? I would think they'd keep the power systems under lock and key, or at least under the control of someone who knew what to do when a UPS starts making noise.
    - jauer 16y ago
      Nope. Data centers are really easy to shut off. Emergency Power Off buttons are often required by code to shut down breakers and UPS. I've heard that this is to prevent loss of life if a tech grabs a livewire and/or let the fire department know that power is removed before they start spraying water.
    - throw_away 16y ago
      The big red buttons are often there as a safety measure in case someone starts getting electrocuted. At very least, though, these very tempting buttons should be labeled in very dire language that they're not to be pressed unless either a) someone is about to die or b) you actually know what you're doing.