13 ms·

Apparently there was a fire in a Datacenter - theres a thread on /r/sysadmin by an insider. https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo_is_dow

Apparently there was a fire in a Datacenter - theres a thread on /r/sysadmin by an insider. https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo_is_down_declining_transactions_and_no/ https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo...

jwcooper 8y ago
Local fire Department commented on their facebook page [1] that dust from construction triggered the fire suppression system and not a fire. [1] https://www.facebook.com/LakeJoFD https://www.facebook.com/LakeJoFD
- bitxbitxbitcoin 8y ago
  The next worst thing for electronics after fire - water! Or are there non liquid fire suppression systems such as pulling all the oxygen out of a room (if there aren't any people in it)?
  - ptr 8y ago
    Yes water isn't typically used in datacenters, halon gas can be used for example. But the noise of the nozzles releasing the gas can actually damage disk drives.
    - xdlachx 8y ago
      yeah, the data center affected apparently had a halon gas suppression system.
    - regecks 8y ago
      There's a nozzle for that :) - http://www.fire-protection.com.au/news/hush-nozzle http://www.fire-protection.com.au/news/hush-nozzle From what I understand, using water/dry pipe isn't unheard of. Some prefer it over Halon - https://blog.equinix.com/blog/2014/03/26/we-must-protect-this-house-against-disaster/ https://blog.equinix.com/blog/2014/03/26/we-must-protect-thi... .
      - SketchySeaBeast 8y ago
        > We think water is superior to using the firefighting chemical compound Halon, because water Is less damaging to technology and Halon can destroy circuit cards. Did not know Halon could destroy circuit cards. Apparently it also damages the Ozone.
        saagarjha 8y ago
        Yeah, some of them are literally CFCs.
  - reaperducer 8y ago
    are there non liquid fire suppression systems such as pulling all the oxygen out of a room Yes. One place I worked (not a tech company, but with tons of electronics), when the fire alarms went off we had xx seconds (I don't remember the number) to get out of the building before something called Intergen was vented into the room to somehow suck all of the oxygen out, and if we were still inside we'd be dead. It must be pretty serious stuff, because we'd have evacuation drills twice a year.
    - deleted 8y ago
      [deleted]
    - deleted 8y ago
      [deleted]
    - deleted 8y ago
      [deleted]
    - saagarjha 8y ago
      This page suggests that Inergen (unless there is a thing called Intergen that I'm confusing it with?) mixes with air to lower the oxygen content, but still remain breathable: http://www.tyco.no/products/Gaseous-Fire-Suppression/inergen-overview http://www.tyco.no/products/Gaseous-Fire-Suppression/inergen...
      - CydeWeys 8y ago
        It's definitely not good to breathe in though, so you can still see why, for liability reasons, the company would want all employees to evacuate if it's going to be deployed, and leave it to firefighters with SCBA equipment.
      - arkades 8y ago
        The resultant 12% oxygen is 60% of normal atmospheric air. It's still breathable, if you're an average healthy person with healthy cardiac and pulmonary function, sitting at rest - but it's not healthy, it's enough that you're going to start seeing systemic responses. The reduction in partial pressure of oxygen being sucked into the lungs will also cause an increase in the partial pressure of carbon dioxide and inert gases. I think the concern likely comes from: -Folks that are of poor cardiac function are going to be evacuating, meaning increased cardiac demand under stress, while being somewhat oxygen-starved. This could tip some folks into an acute episode that otherwise wouldn't. -Folks that are of poor oxygen function, who are borderline hypoxemic to begin with. Think folks with chronic obstructive pulmonary disorder: about 5% of your employees aged 55-65 will have it. You won't suddenly kill a building full of people. I'm guessing the evacuation rush is to make sure they're not liable for unnecessarily sending a couple to the hospital.
  - yread 8y ago
    Halon is also not great - it can damage disks (360 psi) and even if it doesn't it voids the warranty on everything so you'll be getting new hardware
  - shagie 8y ago
    Ahh... The VAX war. http://bogpeople.com/funny/content/vaxwarstory.html http://bogpeople.com/funny/content/vaxwarstory.html > VAXen, my children, just don't belong some places. Suffice to say... yes. There are fire suppression systems that will pull oxygen out of the room. People are advised to leave the room before the fire suppression system takes effect.
    - james_s_tayler 8y ago
      People are also advised to leave the room before the fire takes effect.
mabbo 8y ago
Incredible. They had all their mission-critical infrastructure in a single data center. How many billions of dollars do they make per year? And they can't afford even a tiny bit of redundancy? If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.
- ModFightsCron 8y ago
  Incredible. You made up a really stupid and (for a bank) illegal thing, then said it's stupid! If I were a company, I'd use this as a sign you are not competent enough to receive our money. No dipshit. They have active->sync->async or star replication, live every other bank. Guess what - when you actually fails over it's not always perfectly smooth. The OP made up that "all of wells fargo" went offline, when in reality tiny bits and pieces die. Have you ever worked or did you wander off the high school tech subreddit? When you actually fail over, sometimes not everything fails over smoothly and you have to do database media recovery and replay redo logs.
- barkingcat 8y ago
  I don't think you need this as a sign it's not technically competent. There are other much larger markers.
- mywittyname 8y ago
  Lots of old companies are like this. They don't invest in critical infrastructure because it would bring them below their 20% margins.
  - nostrademons 8y ago
    Wells is weird though. When I started banking with them in 2009, they were ahead of the curve in online infrastructure - it was much easier to access them online than any of the banks I was used to back in Massachusetts. They also had a reputation for being both honest and conservative - they were the only AAA-rated bank in 2007, they were #1 in green rankings, and Warren Buffett had invested in them because of their sound balance sheet. And then starting around 2010 but rapidly accelerating around 2014, everything about them went to shit. The best explanation I can think of is that John Stumpf is a slash & burn sociopath, juicing the numbers so he can get his 473x-the-median-worker paycheck while ruining the company. He wouldn't be alone in the financial world, but it's a shame that a 150+ year old institution can so rapidly go down the toilet.
carlsborg 8y ago
> No power to any of the network or compute equipment and some failovers did not work as expected. Wonder if it was the switchgear that failed. Amazon uses custom firmware in its switchgears because this happens so often (Superbowl 2013 etc.)
- wapoamspomw 8y ago
  Some great reading on the topic from James Hamilton's blog. https://perspectives.mvdirona.com/2013/02/the-power-failure-seen-around-the-world/ https://perspectives.mvdirona.com/2013/02/the-power-failure-... https://perspectives.mvdirona.com/2017/04/at-scale-rare-events-arent-rare/ https://perspectives.mvdirona.com/2017/04/at-scale-rare-even... https://perspectives.mvdirona.com/2017/12/when-you-cant-afford-not-to-have-power-redundancy/ https://perspectives.mvdirona.com/2017/12/when-you-cant-affo...
sys_64738 8y ago
Why didn't it failover to their backup datacenter in another geo?
- ModFightsCron 8y ago
  It did. The claim of the post is false.
- charlietango92 8y ago
  right? DRBC trial by fire, I guess (pardon the pun).

jwcooper 8y ago

Local fire Department commented on their facebook page [1] that dust from construction triggered the fire suppression system and not a fire. [1] https://www.facebook.com/LakeJoFD https://www.facebook.com/LakeJoFD

bitxbitxbitcoin 8y ago
The next worst thing for electronics after fire - water! Or are there non liquid fire suppression systems such as pulling all the oxygen out of a room (if there aren't any people in it)?
- ptr 8y ago
  Yes water isn't typically used in datacenters, halon gas can be used for example. But the noise of the nozzles releasing the gas can actually damage disk drives.
  - xdlachx 8y ago
    yeah, the data center affected apparently had a halon gas suppression system.
  - regecks 8y ago
    There's a nozzle for that :) - http://www.fire-protection.com.au/news/hush-nozzle http://www.fire-protection.com.au/news/hush-nozzle From what I understand, using water/dry pipe isn't unheard of. Some prefer it over Halon - https://blog.equinix.com/blog/2014/03/26/we-must-protect-this-house-against-disaster/ https://blog.equinix.com/blog/2014/03/26/we-must-protect-thi... .
    - SketchySeaBeast 8y ago
      > We think water is superior to using the firefighting chemical compound Halon, because water Is less damaging to technology and Halon can destroy circuit cards. Did not know Halon could destroy circuit cards. Apparently it also damages the Ozone.
      - saagarjha 8y ago
        Yeah, some of them are literally CFCs.
- reaperducer 8y ago
  are there non liquid fire suppression systems such as pulling all the oxygen out of a room Yes. One place I worked (not a tech company, but with tons of electronics), when the fire alarms went off we had xx seconds (I don't remember the number) to get out of the building before something called Intergen was vented into the room to somehow suck all of the oxygen out, and if we were still inside we'd be dead. It must be pretty serious stuff, because we'd have evacuation drills twice a year.
  - deleted 8y ago
    [deleted]
  - deleted 8y ago
    [deleted]
  - deleted 8y ago
    [deleted]
  - saagarjha 8y ago
    This page suggests that Inergen (unless there is a thing called Intergen that I'm confusing it with?) mixes with air to lower the oxygen content, but still remain breathable: http://www.tyco.no/products/Gaseous-Fire-Suppression/inergen-overview http://www.tyco.no/products/Gaseous-Fire-Suppression/inergen...
    - CydeWeys 8y ago
      It's definitely not good to breathe in though, so you can still see why, for liability reasons, the company would want all employees to evacuate if it's going to be deployed, and leave it to firefighters with SCBA equipment.
    - arkades 8y ago
      The resultant 12% oxygen is 60% of normal atmospheric air. It's still breathable, if you're an average healthy person with healthy cardiac and pulmonary function, sitting at rest - but it's not healthy, it's enough that you're going to start seeing systemic responses. The reduction in partial pressure of oxygen being sucked into the lungs will also cause an increase in the partial pressure of carbon dioxide and inert gases. I think the concern likely comes from: -Folks that are of poor cardiac function are going to be evacuating, meaning increased cardiac demand under stress, while being somewhat oxygen-starved. This could tip some folks into an acute episode that otherwise wouldn't. -Folks that are of poor oxygen function, who are borderline hypoxemic to begin with. Think folks with chronic obstructive pulmonary disorder: about 5% of your employees aged 55-65 will have it. You won't suddenly kill a building full of people. I'm guessing the evacuation rush is to make sure they're not liable for unnecessarily sending a couple to the hospital.
- yread 8y ago
  Halon is also not great - it can damage disks (360 psi) and even if it doesn't it voids the warranty on everything so you'll be getting new hardware
- shagie 8y ago
  Ahh... The VAX war. http://bogpeople.com/funny/content/vaxwarstory.html http://bogpeople.com/funny/content/vaxwarstory.html > VAXen, my children, just don't belong some places. Suffice to say... yes. There are fire suppression systems that will pull oxygen out of the room. People are advised to leave the room before the fire suppression system takes effect.
  - james_s_tayler 8y ago
    People are also advised to leave the room before the fire takes effect.

mabbo 8y ago

Incredible. They had all their mission-critical infrastructure in a single data center. How many billions of dollars do they make per year? And they can't afford even a tiny bit of redundancy? If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.

ModFightsCron 8y ago
Incredible. You made up a really stupid and (for a bank) illegal thing, then said it's stupid! If I were a company, I'd use this as a sign you are not competent enough to receive our money. No dipshit. They have active->sync->async or star replication, live every other bank. Guess what - when you actually fails over it's not always perfectly smooth. The OP made up that "all of wells fargo" went offline, when in reality tiny bits and pieces die. Have you ever worked or did you wander off the high school tech subreddit? When you actually fail over, sometimes not everything fails over smoothly and you have to do database media recovery and replay redo logs.
barkingcat 8y ago
I don't think you need this as a sign it's not technically competent. There are other much larger markers.
mywittyname 8y ago
Lots of old companies are like this. They don't invest in critical infrastructure because it would bring them below their 20% margins.
- nostrademons 8y ago
  Wells is weird though. When I started banking with them in 2009, they were ahead of the curve in online infrastructure - it was much easier to access them online than any of the banks I was used to back in Massachusetts. They also had a reputation for being both honest and conservative - they were the only AAA-rated bank in 2007, they were #1 in green rankings, and Warren Buffett had invested in them because of their sound balance sheet. And then starting around 2010 but rapidly accelerating around 2014, everything about them went to shit. The best explanation I can think of is that John Stumpf is a slash & burn sociopath, juicing the numbers so he can get his 473x-the-median-worker paycheck while ruining the company. He wouldn't be alone in the financial world, but it's a shame that a 150+ year old institution can so rapidly go down the toilet.

carlsborg 8y ago

> No power to any of the network or compute equipment and some failovers did not work as expected. Wonder if it was the switchgear that failed. Amazon uses custom firmware in its switchgears because this happens so often (Superbowl 2013 etc.)

wapoamspomw 8y ago
Some great reading on the topic from James Hamilton's blog. https://perspectives.mvdirona.com/2013/02/the-power-failure-seen-around-the-world/ https://perspectives.mvdirona.com/2013/02/the-power-failure-... https://perspectives.mvdirona.com/2017/04/at-scale-rare-events-arent-rare/ https://perspectives.mvdirona.com/2017/04/at-scale-rare-even... https://perspectives.mvdirona.com/2017/12/when-you-cant-afford-not-to-have-power-redundancy/ https://perspectives.mvdirona.com/2017/12/when-you-cant-affo...

sys_64738 8y ago

Why didn't it failover to their backup datacenter in another geo?

ModFightsCron 8y ago
It did. The claim of the post is false.
charlietango92 8y ago
right? DRBC trial by fire, I guess (pardon the pun).