5 ms·

I get the feeling that national disasters are actually the easy thing to anticipate. If you are on the coast, hurricane or flooding. North Dakota is tornadoes

I get the feeling that national disasters are actually the easy thing to anticipate. If you are on the coast, hurricane or flooding. North Dakota is tornadoes and flooding in river basins. Fire is an everyone problem. Its the weird crap that really gets your goat. I remember finding out the power main to our current place is located next to a T-section of road. Slide forward too far and hit the power main. Brilliant placement. Or when the primary, cannot fail contingency fails. A place I once worked at had a >$1 Million UPS (generator / batteries lasting weeks) for their data center. Semi hit the power main, and the UPS failed to turn on. I actually got wrote up[1] as being "paranoid" in my backup strategy for a government grant we had in the 90s. The joke of it was that the site visitor who wrote me up was a project director at another site. That site experienced a flood and required extensive efforts to recreate their data. I was young, and a bit vengeful and might have made some comments during the next national meetup. Sometimes, a lot of disaster recovery and contingency planning is politics above the level of the people who should know better. 1) They thought it was a problem during a site visit and we had to write a response that had to get approved.

jacquesm 9y ago
> I get the feeling that national disasters are actually the easy thing to anticipate. Their presence, yes. But the frequency and intensity are hard to put any bounds on. > I remember finding out the power main to our current place is located next to a T-section of road. Slide forward too far and hit the power main. Brilliant placement. Something exactly like that happened with a gas line in Slupsk, Poland. The effects were incredible. > A place I once worked at had a >$1 Million UPS (generator / batteries lasting weeks) for their data center. Semi hit the power main, and the UPS failed to turn on. Just having a generator is not enough, you need batteries for the ride-through period and you need to do periodic tests. Maybe they did and it still didn't turn on? I'd take being called paranoid as a badge of honor, better to be paranoid and up and running than down and searching the yellow pages for data recovery services.
- protomyth 9y ago
  Oh, they had the whole line. UPS in the server room, with batteries and diesel engine outside. Tested quarterly, I was told. In server room batteries did their job, but the stuff outside didn't. Took 4 hours to restore power and another 32 hours to restart the servers and verify their data. [edit: about the site visit calling me paranoid] I was a bit ticked off, but I guess someone who didn't even do the basics, as we later found out, would not get the need.
  - jacquesm 9y ago
    Ok. Even a small DC that I'm intimately familiar with has a back-up diesel that they test religiously on the 1st of every month, they have the whole building running on the genset for an hour or two and then they shut it all down again. Iirc it failed to start twice in the years that this setup has been there, which means the chances of a problem will never be quite '0'. When I ran a gas station in North America one of the first things I did was install a diesel generator. My reasoning was that with on average 50K liters of diesel in the underground tanks and the genset powering the pumps it should be good for most problems that we'd face. And in fact it was, which we managed to prove beyond any doubt when all of the North-West of the US and a good chunk of Canada went without power for a couple of days within a year of installing that genset. Sometimes being prepared is a waste of money and time in retrospect, but it helps you sleep well, and sometimes it is the difference between a community continuing to function or people dying.
    - Animats 9y ago
      The classic Bell System standard was that generators were started once a month and run for several hours. Once a year, each central office ran on emergency power for 24 hours, with external power shut off. In the entire history of the Bell System prior to electronic switching, no central office was ever offline for more than 30 minutes for any reason other than a natural disaster.
      - MandieD 9y ago
        Of course I've known that a DR plan isn't valid until you've tested it against your production environment, but I'd never heard that Bell System story before. It's inspirational - run 24 hours in emergency mode each year, with the normal way turned off/disconnected, don't just do failover/failback. Thanks for bringing it up!
- vacri 9y ago
  > down and searching the yellow pages for data recovery services. Back when I was a medical tech, we lost a hard drive with patient data on it. I ran a chkdsk with no luck, and out of ideas, called a data recovery company. They stated it was $3-5k to attempt to recover, and asked if we'd done anything to the drive. "Just a chkdsk, nothing else". The guy on the other end of the phone was genuinely annoyed by this. "I wish you people wouldn't do that" delivered with that tone. Yes, this guy's position was that if you ever have a disk problem, your first port of call is to drop $3k+ at a data recovery place, don't even attempt to do some preliminary checking.
  - jacquesm 9y ago
    I can see his point though. From his point of view, by the time the disks arrive on his desk anything that tried to repair the damage has likely made it worse and hence his job that much harder. But of course $3-5k is sufficient money that most companies will make the first attempt themselves, especially since the alternative does not make any guarantees and will cost money either way.
    - paulmd 9y ago
      I've had to do some data recovery during the course of my personal life. If the data is valuable, the #1 rule is "always run a dd backup to another device first" IMO. Then you can play around on a device that you know isn't on the brink of failing.
      - deleted 9y ago
        [deleted]
aidos 9y ago
A place I once worked had their servers in-house. They had UPS on the roof, duel everything (fibre links etc). Apparently there wasn't a single point of failure in the system. One day, the servers just vanish. We go into the server room to check. Someone had plugged all the machines into the same multi box / power extension cable and the fuse in it had popped. Sometimes all your planning can be undone - especially once humans are involved :-)
- digi_owl 9y ago
  Makes me think of the story where the cleaning people would unplug a server rack to power their carpet cleaner.
  - joejerryronnie 9y ago
    Haha, happened to me! Didn't see this until after I posted a reply to the parent :)
    - digi_owl 9y ago
      Heh. The story i recall their initial problem was that the failure as always after office hours, but they could not find any fault with the hardware or software. So as a last desperate attempt, the person telling it basically sat watch inside the server room and noticed the door opening, a arm reaching in, unplugging a socket near the door, and plugging in a cable from the hallway. The very next day security was switching the lock on that door.
- jacquesm 9y ago
  > Apparently there wasn't a single point of failure in the system. Redundant power supplies on alternate phases is a pretty common element in designing any kind of server setup. So this wasn't a setup with a 'single point of failure', it was a setup with a very explicit single point of failure (several, in fact). Someone linked to this excellent slide deck yesterday: https://c3.nasa.gov/dashlink/static/media/other/ObservedFailures1.html https://c3.nasa.gov/dashlink/static/media/other/ObservedFail... Worth an hour of your time, not to make you paranoid but more to make it plain how hard this sort of planning really is.
hexane360 9y ago
>Fire is an everyone problem. That's not even remotely true. This is another one of those weird HN tunnel visions. http://lakedillonfirerescue.org/wp-content/uploads/2013/03/wildfire-risk-potential.bmp http://lakedillonfirerescue.org/wp-content/uploads/2013/03/w...
- protomyth 9y ago
  Last I checked, everybody's data center or building can catch fire or have a fire inside. I would love to see the building that has computer equipment in it that cannot catch on fire. You are one bad UPS / power supply away from a rack fire. Never mind the electrical panel itself.
  - ghaff 9y ago
    The context was natural disasters though--so pretty much wildfires.
    - protomyth 9y ago
      And every place even those listed on the map can get them unless you build your data center on some terrain with absolutely no plan life around. You need to deal with fire outbreaks even if you are not on the map.
      - jacquesm 9y ago
        And even then there is lightning induced fire. Plenty of natural fires start that way and there is no reason why your building can't be the starting point of one. Though fortunately most DCs take lightning protection quite serious and are probably amongst the harder buildings to ignite.
        sqeaky 9y ago
        Lightning Rods can prevent this.
joejerryronnie 9y ago
Years ago we had a small, in house, departmental server room at Stanford. It was by no means industrial strength but we did our best on a limited budget (security, climate control, battery backups, disaster recovery procedures, etc). One night we all got pinged about impending server failures with 30 minutes to respond before the UPS ran out of juice. Turns out the cleaning crew needed an outlet for their vacuum cleaner and decided to unplug one rack of servers to finish up their shift. This episode reminds me of a favorite haiku: We think about risks. We have contingency plans. Oops…but not for that.

jacquesm 9y ago

> I get the feeling that national disasters are actually the easy thing to anticipate. Their presence, yes. But the frequency and intensity are hard to put any bounds on. > I remember finding out the power main to our current place is located next to a T-section of road. Slide forward too far and hit the power main. Brilliant placement. Something exactly like that happened with a gas line in Slupsk, Poland. The effects were incredible. > A place I once worked at had a >$1 Million UPS (generator / batteries lasting weeks) for their data center. Semi hit the power main, and the UPS failed to turn on. Just having a generator is not enough, you need batteries for the ride-through period and you need to do periodic tests. Maybe they did and it still didn't turn on? I'd take being called paranoid as a badge of honor, better to be paranoid and up and running than down and searching the yellow pages for data recovery services.

protomyth 9y ago
Oh, they had the whole line. UPS in the server room, with batteries and diesel engine outside. Tested quarterly, I was told. In server room batteries did their job, but the stuff outside didn't. Took 4 hours to restore power and another 32 hours to restart the servers and verify their data. [edit: about the site visit calling me paranoid] I was a bit ticked off, but I guess someone who didn't even do the basics, as we later found out, would not get the need.
- jacquesm 9y ago
  Ok. Even a small DC that I'm intimately familiar with has a back-up diesel that they test religiously on the 1st of every month, they have the whole building running on the genset for an hour or two and then they shut it all down again. Iirc it failed to start twice in the years that this setup has been there, which means the chances of a problem will never be quite '0'. When I ran a gas station in North America one of the first things I did was install a diesel generator. My reasoning was that with on average 50K liters of diesel in the underground tanks and the genset powering the pumps it should be good for most problems that we'd face. And in fact it was, which we managed to prove beyond any doubt when all of the North-West of the US and a good chunk of Canada went without power for a couple of days within a year of installing that genset. Sometimes being prepared is a waste of money and time in retrospect, but it helps you sleep well, and sometimes it is the difference between a community continuing to function or people dying.
  - Animats 9y ago
    The classic Bell System standard was that generators were started once a month and run for several hours. Once a year, each central office ran on emergency power for 24 hours, with external power shut off. In the entire history of the Bell System prior to electronic switching, no central office was ever offline for more than 30 minutes for any reason other than a natural disaster.
    - MandieD 9y ago
      Of course I've known that a DR plan isn't valid until you've tested it against your production environment, but I'd never heard that Bell System story before. It's inspirational - run 24 hours in emergency mode each year, with the normal way turned off/disconnected, don't just do failover/failback. Thanks for bringing it up!
vacri 9y ago
> down and searching the yellow pages for data recovery services. Back when I was a medical tech, we lost a hard drive with patient data on it. I ran a chkdsk with no luck, and out of ideas, called a data recovery company. They stated it was $3-5k to attempt to recover, and asked if we'd done anything to the drive. "Just a chkdsk, nothing else". The guy on the other end of the phone was genuinely annoyed by this. "I wish you people wouldn't do that" delivered with that tone. Yes, this guy's position was that if you ever have a disk problem, your first port of call is to drop $3k+ at a data recovery place, don't even attempt to do some preliminary checking.
- jacquesm 9y ago
  I can see his point though. From his point of view, by the time the disks arrive on his desk anything that tried to repair the damage has likely made it worse and hence his job that much harder. But of course $3-5k is sufficient money that most companies will make the first attempt themselves, especially since the alternative does not make any guarantees and will cost money either way.
  - paulmd 9y ago
    I've had to do some data recovery during the course of my personal life. If the data is valuable, the #1 rule is "always run a dd backup to another device first" IMO. Then you can play around on a device that you know isn't on the brink of failing.
    - deleted 9y ago
      [deleted]

aidos 9y ago

A place I once worked had their servers in-house. They had UPS on the roof, duel everything (fibre links etc). Apparently there wasn't a single point of failure in the system. One day, the servers just vanish. We go into the server room to check. Someone had plugged all the machines into the same multi box / power extension cable and the fuse in it had popped. Sometimes all your planning can be undone - especially once humans are involved :-)

digi_owl 9y ago
Makes me think of the story where the cleaning people would unplug a server rack to power their carpet cleaner.
- joejerryronnie 9y ago
  Haha, happened to me! Didn't see this until after I posted a reply to the parent :)
  - digi_owl 9y ago
    Heh. The story i recall their initial problem was that the failure as always after office hours, but they could not find any fault with the hardware or software. So as a last desperate attempt, the person telling it basically sat watch inside the server room and noticed the door opening, a arm reaching in, unplugging a socket near the door, and plugging in a cable from the hallway. The very next day security was switching the lock on that door.
jacquesm 9y ago
> Apparently there wasn't a single point of failure in the system. Redundant power supplies on alternate phases is a pretty common element in designing any kind of server setup. So this wasn't a setup with a 'single point of failure', it was a setup with a very explicit single point of failure (several, in fact). Someone linked to this excellent slide deck yesterday: https://c3.nasa.gov/dashlink/static/media/other/ObservedFailures1.html https://c3.nasa.gov/dashlink/static/media/other/ObservedFail... Worth an hour of your time, not to make you paranoid but more to make it plain how hard this sort of planning really is.

hexane360 9y ago

>Fire is an everyone problem. That's not even remotely true. This is another one of those weird HN tunnel visions. http://lakedillonfirerescue.org/wp-content/uploads/2013/03/wildfire-risk-potential.bmp http://lakedillonfirerescue.org/wp-content/uploads/2013/03/w...

protomyth 9y ago
Last I checked, everybody's data center or building can catch fire or have a fire inside. I would love to see the building that has computer equipment in it that cannot catch on fire. You are one bad UPS / power supply away from a rack fire. Never mind the electrical panel itself.
- ghaff 9y ago
  The context was natural disasters though--so pretty much wildfires.
  - protomyth 9y ago
    And every place even those listed on the map can get them unless you build your data center on some terrain with absolutely no plan life around. You need to deal with fire outbreaks even if you are not on the map.
    - jacquesm 9y ago
      And even then there is lightning induced fire. Plenty of natural fires start that way and there is no reason why your building can't be the starting point of one. Though fortunately most DCs take lightning protection quite serious and are probably amongst the harder buildings to ignite.
      - sqeaky 9y ago
        Lightning Rods can prevent this.

joejerryronnie 9y ago

Years ago we had a small, in house, departmental server room at Stanford. It was by no means industrial strength but we did our best on a limited budget (security, climate control, battery backups, disaster recovery procedures, etc). One night we all got pinged about impending server failures with 30 minutes to respond before the UPS ran out of juice. Turns out the cleaning crew needed an outlet for their vacuum cleaner and decided to unplug one rack of servers to finish up their shift. This episode reminds me of a favorite haiku: We think about risks. We have contingency plans. Oops…but not for that.