10 ms·

Something is burning in the server room; how can I quickly identify what it is?

200 points by usea 14y ago

lotsofcows 14y ago
Can of deodorant.
yardie 14y ago
You can tell from the responses who has a real DR plan and who are just winging it. The DR-backed commenters can switch to site B with nary a worry. Everyone else is trying to justify keeping the server room running while something is burning. To them a misplaced backhoe is a bigger problem than you know a server burning. I love the sanity check part. But really, you're keeping someone on hand to drag your ass out after you pass out while you sniff up toxic fumes.
- deleted 14y ago
  [deleted]
- growse 14y ago
  A million times this. If turning a whole datacenter off is catastrophic for your business, you've not done your risk management properly. Single events that take down datacenters happen. Not being prepared for them is unforgivable.
  - rdl 14y ago
    Even with DR, it has a cost. Especially if you're a colo provider, powering off your whole facility (or, ideally, at least a room) to find smoke is going to have cost, even if all your customers have DR plans. In most datacenters I've seen, I'd probably be willing to do a run through with IR cam/temp probe, or just visual inspection, with a handheld 1211, especially if I had a respirator, if it were just "smell of smoke". Clear view and path to two exits, someone at the EPO switch, etc. The "big scary things" are battery plant and generator plant, and any kind of subfloor or ductwork. As long as the fire isn't in any of those, it's far less of a big deal. I probably wouldn't EPO a room for a server on fire, either -- just kill the rack, which takes slightly longer. I've been in places where "smell of smoke" was a fucknozzle smoking a cigar or burning leaves outdoors outside an air intake, and another where it was a smoker's coat being put on an air handler.
    - dredmorbius 14y ago
      But it happens. Our colo went down. Fire system triggered it (no actual fire was harmed in the triggering, however). On Thanksgiving. When I'd volunteered for on-call seeing as my co-worker had family to attend to and I didn't. He thanked me afterward. Our cage was reasonably straightforward to bring up, once power was restored. The colo facility as a whole took a few days to bring all systems up, apparently some large storage devices really don't like being Molly-switched.
      - rdl 14y ago
        Yeah. So estimate the cost there are like $100-250k? I'm willing to accept a pretty low risk to my life for ~15 minutes of searching to save my company $250k. It's a risk on the order of riding a motorcycle from Oakland to San Jose in rush hour, I'd roughly estimate.
        yardie 14y ago
        "I can totally reach into the back of this gigantic, flesh eating machine to move that widget a little bit to the right. Management might give me a raise for saving them money!" -famous last words of a former factory worker.
        rdl 14y ago
        "Do not look into laser with remaining eye" is the classic, though.
        reeses 14y ago
        Enjoy that cancer from burning and/or airborne plastic/arsenic/mystery material in 20 years. And which part of Oakland? That can be a pretty broad range of risk. :-)
        rdl 14y ago
        "You wouldn't download a new lung, would you?" (RIAA ad in 2033). I lived in a tent 200m downwind of a 24x7 burning trash dump on a former Iraqi and then USAF military base, for some time, so I think I've got particulate risk checked already.
  - protomyth 14y ago
    I was working at a place as a contractor and they had an amazing backup power system (expensive, diesel with batteries). Semi hits the power main outside the server room and for some reason the diesel never starts. Whole datacenter loses power in under 15 minutes. Always plan for the single event because no amount of money will keep a single site running.
    - tomjen3 14y ago
      The biggest danish ISP lost service for quite some time because the truck that were to deliver their new emergency power crashed into the mains transformer stations. Of course they are idiots, they should never have powered their own mains of before they got the new one installed. But it was still pretty funny.
  - CamperBob2 14y ago
    Counterpoint: As a customer, I don't want to pay for the kind of redundancy that encourages employees to shut down the whole data center when a single tantalum cap pops in a power supply somewhere. The fact is, there aren't that many flammable things in a data center. Nobody is going to die because they wandered up and down the aisles after they "thought they smelled something burning," in the absence of any visible smoke. The guidelines in the highest-voted answer on the SO page make sense to me. 1: If you actually see smoke or fire in any significant amount, evacuate. 2: Make someone else aware of what's going on before doing anything else. 3: Keep your escape options open. 4: Think about how much time you can safely spend "guessing", and don't exceed it. 5: Don't second-guess your own common sense. You aren't paid to be a fireman or a hero.
  - ghshephard 14y ago
    Almost nobody (for any reasonable definition of nobody) has immediate redundancy for their data center. I've worked for three $1B+ companies whose entire business was based on their data center being up and running - and none of them would have returned to service in less than 48 hours if a data center had gone down. "Not being prepared for them is unforgivable" - would mean that 99% of business do not deserve forgiveness. It just doesn't make sense to have that kind of redundancy for such a rare event for all but a very, very small minority of businesses. (Telecoms, Google, Stock Exchange, 911, etc...)
- Aloisius 14y ago
  I'm not sure immediately pressing the big red button because you mistook the smell a lead acid battery out gassing for something burning (with no smoke) is necessarily a measure of preparedness.
ck2 14y ago
Maybe we need a diy thermal sensor that plugs into an android or iphone device? Oh wow, it exists: http://www.instructables.com/id/Thermal-Imaging-Phone-Camera/ http://www.instructables.com/id/Thermal-Imaging-Phone-Camera... http://www.robhopeless.com/search/label/Thermal%20Imaging http://www.robhopeless.com/search/label/Thermal%20Imaging http://www.kickstarter.com/projects/andyrawson/ir-blue-thermal-imaging-smartphone-accessory http://www.kickstarter.com/projects/andyrawson/ir-blue-therm... http://rh-workshop-llc.myshopify.com/ http://rh-workshop-llc.myshopify.com/ Costs only $150 to make? Open-source: https://github.com/RHWorkshop/ https://github.com/RHWorkshop/
- aw3c2 14y ago
  Wow, I wonder what the range of that is. I.e. if I can use it to analyse the thermal loss of houses.
  - ck2 14y ago
    I believe that's what the inventor originally made it for, if you look at his postings.
  - walshemj 14y ago
    why do you need that you from my termofluids classes you should be able to calculate the heat loss for a given Delta T by taking a ball park figure for the materials ie so many square meters of brick Glass etc.
    - wiml 14y ago
      You use it when you don't know the material (because you're surveying an old house) or you don't know the delta-T (because you're looking for a hidden fire, grow op, etc).
      - NickNameNick 14y ago
        Or you're trying to track down head loss caused by construction flaws, especially leaks and drafts caused by cracks or gaps. Also leaks caused by improperly installed power points. Missing or settled insulation is also common.
hp50g 14y ago
Infra red camera!
rdl 14y ago
IR/thermal imaging cameras are SO USEFUL. I had a fire (bathroom fan caught fire due to being 45y old, knocked it down and extinguished it myself, but was worried about extension in the ceiling/duct). Oakland FD came out and used their IR camera to check the heat from the ceilings nearby. Hilariously they found a hot water pipe (running between bathroom and kitchen) and almost axed the ceiling open (turning $1500 in damage into $3k+), but their captain was smart and figured it out from another angle. Really tempted to hack an EOS 5Dm3 into an IR camera next. Not so much for fires as night vision, but it would be useful for fires too. I'm not sure how useful an IR camera is at detecting heat, since things which aren't yet on fire are not quite so infrared, though. I usually use a Fluke IR temp meter when cooking and to find hot wires/etc. in the datacenter, though.
- Udo 14y ago
  They are useful but they do have limitations. I'm not sure it would have necessarily detected the faulty battery in this case. Last year, there was a fire at my house and the FD searched for the source for three hours. With thermal imaging and everything. It was inside the walls, no open flame, just a lot of smoke and no clear readings on the imager. That was pretty frightening. (However when they finally did find it, they put it out in a couple of minutes.)
  - rdl 14y ago
    I'm glad it motivated me to get ABC Dry and Halon 1211 extinguishers for both rooms and the car, at least. In a "real" datacenter, you should have smoke sensors which would map where heat/smoke is coming from (since you have controlled airflow, it should be obvious which rack or small group of racks was the source -- it doesn't just exhaust into the whole room). But it's pretty clear this wasn't a "real" datacenter by their lack of protocols for handling fire, it was some office server thing.
    - Udo 14y ago
      I'm just saying these things can quickly get more complicated than expected. I too had extinguishers handy at the time but of course I didn't know what to use them on (and neither did the FD for three hours). These electrical fires can be tricky to debug, especially when different kinds of barriers come into play. > But it's pretty clear this wasn't a "real" datacenter by their lack of protocols for handling fire, it was some office server thing. Probably. What's the right protocol though? In this case, it was apparently clear that something minor was amiss, nothing that would justify shutting down the whole thing. In any case, flooding the room with inert gas would probably not have made much of a difference, as it looks like the battery was never actually burning.
iSnow 14y ago
What kind of server room is this which is not equipped with smoke detectors?
- dredmorbius 14y ago
  Depending on how effective the ventilation system is / how diffuse the smoke is, the detectors may not do much to pinpoint where the smoke originated. Many DCs only have so many smoke detectors, at fairly wide intervals. There can be a lot of devices inside a 30' square.
- gambiting 14y ago
  If it's just the "melting plastic" kind of smoke, it won't trigger smoke detectors. And I believe that his battery wasn't actually on fire - if it was, then yes, smoke detectors would have triggered.
  - HeyLaughingBoy 14y ago
    Oh yes, it will. I made my 12 year-old read the story (along with pictures) of a girl his age who was trapped in a burning house after he set off the smoke alarms at midnight by melting bits of plastic in his room. Hopefully he learned something that night.
    - eridius 14y ago
      How did setting off smoke detectors cause a fire?
      - HeyLaughingBoy 14y ago
        Poor sentence structure on my part. My son set off the smoke detectors by melting plastic in his bedroom with a cigarette lighter. To demonstrate the danger of what could come from this, I had him read a news report of a girl who was screwing around with fire and ended up being badly burned over most of her body.
- 14y ago
patio11 14y ago
You ask the fire marshal what was burning, about an hour or so after you've Big Red Button'ed the server room. This story is giving the Japanese engineer in me apoplexy.
- elektronaut 14y ago
  Acting quick and getting a fire under control often makes the difference between an emergency and a disaster. But only after you've made sure the building is being evacuated, and the fire department has been alerted. Don't be afraid to call the emergency number. They'll know what to do and walk you through it. Under no circumstance should you enter a room filled with smoke. Smoke inhalation is incredibly dangerous.
  - MatthewPhillips 14y ago
    Smoke from electronics, especially.
    - kefka 14y ago
      Speaking of consumer electronics: When I went to the Midwest Reprap festival, for some reason power was being a bit flakey. The guy who's sitting across me had 2 printers using ATX power supplies. Suddenly, the PSU sparks out and then outgasses a stream of smoke pillar 4 foot in diameter for the next 2-3 minutes. Near the top of the building perhaps 3 stories up, it looks like a mini-Hiroshima with mushroom cloud going. And that was from a dinky ATX PSU.
      - GFischer 14y ago
        Heh, they once shipped 110v PSUs to us 220v users... I had to build systems at the time, the first one I plugged in made an awesome explosion :P
        ordinary 14y ago
        My first (indeed, only) experience with the smell of burnt human flesh and hair was when a friend of mine flipped the switch on a PSU to 110v when it plugged in to 220v. He was fine, but the PSU didn't make it. RIP.
JimmaDaRustla 14y ago
Lead acid batteries will have a rotten smell after shorting out - more so in wet ones, but SLA will smell the same. If you wait to the point that an SLA smells, it has probably expanded and caused internal damage to your UPS/server rack, albeit minor if you can manage to get it out without dismantling anything.
jahabrewer 14y ago
It's been a while since I've been in IT, but don't failing UPS batteries put off fumes that destroy your lungs? A friend of a friend was a hero and shutdown his datacenter cleanly/recovered some hard drives during a situation like this. He got severe lung damage (not from fire).
whizzkid 14y ago
I couldn't answer the question there because of reputation thing. (just signed up) Here is a little bit different approach; If you are in the room and smelled the burn, that means something is already happened and you are dealing with its result and possible side effects, and that gives you possibly enough time before shutting down everything or getting out of the room. Your chance of not being harmed by this situation is high at least for 5-10 minutes. In this case, having a termal check would not help you a lot since burned hardware is most probably not functioning anymore and might be colder than the regular servers. The other option would be that it is still working but not causing any fire yet so heat is not much different than the usual. Now, smell is your only evidence, I am hoping you guys have air conditioner in the server room. Put it on the max level so that the smell will not be so strong everywhere. This can help you identify where the smell is coming from. Before you check the smell, get out of the server room, breath as much as fresh air you can, so that your realisation will be sharper when you get back to server room. Having your colleague with makes the process faster. This would be my first reaction to these kind of situations. It is of course costly to turn off whole system but don't forget that it is not important than you!
jeremyjh 14y ago
I'm not sure if everyone responding really read this all that carefully. There was absolutely no mention of smoke in this question. There was a "smell". If you drop an entire datacenter, you are easily looking at $100K+ in damages just to reset the room. So, getting a buddy and taking some time to look for the problem until it is found or until you actually are seeing smoke or other specific danger seems like a pretty reasonable course of action.
- DanBC 14y ago
  X times out of Y that's going to be a reasonable course of action. But one time someone will die, and at that point (because it's rare and we freak out at rare dangers) people will be up in arms about it, and about how stupid and irresponsible it is to not hit the button.
  - ttrreeww 14y ago
    ...and that someone could be you.
  - johngalt 14y ago
    Panic can be dangerous as well. A halon dump can asphyxiate someone who doesn't reach the oxygen mask or an exit in time. Dropping a whole server room without seeing any smoke or fire is silly. Do you pull a fire alarm if someone smokes a cigarette indoors?
blantonl 14y ago
It is pretty simple. You call the fire department. They have a TIC (thermal imaging camera) that can detect heat/overheating sources pretty quickly. Plus, it's kind of nice to have them on hand in case a smell progresses to a fire.
- micro-ram 14y ago
  I agree. Even if you call just to alert them that something is not right. Let them roll one truck just to have someone on hand in case someone gets burnt or shocked.
- konstruktor 14y ago
  I agree. People seem to imagine that all they do is put out fires with a hose. They are professionals who know how to manage potentially hazardous situations.
peterwwillis 14y ago
HVAC. Has no noticeable smoke (it'd probably be outside the building anyway) but pumps a burning smell into the room when the motors start dying or aren't oiled right. Don't hit the big red button just because you smell burning.
dattaway 14y ago
Lead based batteries often have a usable life of 3-5 years. Chances are, the others of that vintage are already dry and have already failed. Then they will rupture, often with smoke as their series connected brothers try to push electrons.
- micro-ram 14y ago
  I have seen plenty of UPS batteries swell up so big they can't be removed without disassembly. The only indication was the failed self test. OP did say it was the UPS in the rack next to his production DB. Don't forget Capacitor Plague. I still see it regularly. https://en.wikipedia.org/wiki/Capacitor_plague https://en.wikipedia.org/wiki/Capacitor_plague Have a plan, be safe.
protomyth 14y ago
I've had this happen to me once. No alerts and all boxes were up, but there was a smell in the room. I went machine by machine and UPS by UPS and nothing was wrong or burning(1). Next day we find out the breaker panel next door had a short that blew out several breakers. Smell was vented into the server room. So, not always your room, could be something else just as or more dangerous. 1) shut down all machines, unplug all UPSes, open every case
squozzer 14y ago
Temperature indicator stickers.
- stevenrace 14y ago
  At $5+/ea, that quickly adds up and difficult to parse in large numbers in duress. Whereas an IR camera is rather effective and perhaps cheaper.
darwinGod 14y ago
For some more context, this was the OP's previous question in Stackoverflow. http://serverfault.com/questions/420877/ive-inherited-a-rats-nest-of-cabling-what-now http://serverfault.com/questions/420877/ive-inherited-a-rats... Doesn't that change the entire question!
- astrodust 14y ago
  If there was a burning smell in that, yikes, hit the Big Red Button and turn in your resignation.
  - micro-ram 14y ago
    Seeing that rats nest I would have let it burn!
FireBeyond 14y ago
If it's that important, buy a TIC (thermal imaging camera). They can be had for under $10,000 and will show you actual hotspots. Walk through, sweeping every item.
lobster45 14y ago
Thanks for this story. I am ordering a fire extinguisher for our server room now.
snarfy 14y ago
It helps to prioritize what you look for. 9 times out of 10 it's power related.
ChuckMcM 14y ago
I spent a couple of summers interning at IBM and one of the things they taught you in orientation was the sound of "imminent halon dump" (the alarm that said Halon was about to be used in the machine room). The instructions were, hold your breath and make for an exit immediately. Failure to do so would lead to asphyxiation.
johngalt 14y ago
'hmmm... probably UPS battery venting' click 'yup' When UPS batteries vent it has a distinctive odor. It's very pungent and sulfuric, but it doesn't smell like a fire or melting silicon. Any experienced operations guy has smelled it before. Additionally in most fire suppression systems the Big Red Button is the abort button. A well designed room will dump itself when it detects smoke after a short evacuation alarm. It's precisely designed to keep people from screwing around with a real fire. They must make the active decision to stop fire suppression rather than start fire suppression.
chris_wot 14y ago
Just an idea: the server room should be segmented into smaller areas with isolated power circuits. Using the sniff test, if you are truly concerned that you are about to have a fire, then it's only responsible to start an orderly powerdown to prevent equipment loss, and more importantly, prevent injuries to people. If you start shutting down areas of the datacentre that appear to be closest to the smoke, then you will have a better chance of locating the issue in the fastest possible timeframe, with minimum disruption. On top of this, if you then have critical infrastructure that you must keep running, then you keep your failover servers in different areas and failover to that equipment. I'm not a server or datacentre guy in any way, but doesn't this seem sensible?
Qantourisc 14y ago
Get in an electronics "expert" they know what component smells like what :)

200 points by usea 14y ago

Can of deodorant.

You can tell from the responses who has a real DR plan and who are just winging it. The DR-backed commenters can switch to site B with nary a worry. Everyone else is trying to justify keeping the server room running while something is burning. To them a misplaced backhoe is a bigger problem than you know a server burning. I love the sanity check part. But really, you're keeping someone on hand to drag your ass out after you pass out while you sniff up toxic fumes.

deleted 14y ago
[deleted]
growse 14y ago
A million times this. If turning a whole datacenter off is catastrophic for your business, you've not done your risk management properly. Single events that take down datacenters happen. Not being prepared for them is unforgivable.
- rdl 14y ago
  Even with DR, it has a cost. Especially if you're a colo provider, powering off your whole facility (or, ideally, at least a room) to find smoke is going to have cost, even if all your customers have DR plans. In most datacenters I've seen, I'd probably be willing to do a run through with IR cam/temp probe, or just visual inspection, with a handheld 1211, especially if I had a respirator, if it were just "smell of smoke". Clear view and path to two exits, someone at the EPO switch, etc. The "big scary things" are battery plant and generator plant, and any kind of subfloor or ductwork. As long as the fire isn't in any of those, it's far less of a big deal. I probably wouldn't EPO a room for a server on fire, either -- just kill the rack, which takes slightly longer. I've been in places where "smell of smoke" was a fucknozzle smoking a cigar or burning leaves outdoors outside an air intake, and another where it was a smoker's coat being put on an air handler.
  - dredmorbius 14y ago
    But it happens. Our colo went down. Fire system triggered it (no actual fire was harmed in the triggering, however). On Thanksgiving. When I'd volunteered for on-call seeing as my co-worker had family to attend to and I didn't. He thanked me afterward. Our cage was reasonably straightforward to bring up, once power was restored. The colo facility as a whole took a few days to bring all systems up, apparently some large storage devices really don't like being Molly-switched.
    - rdl 14y ago
      Yeah. So estimate the cost there are like $100-250k? I'm willing to accept a pretty low risk to my life for ~15 minutes of searching to save my company $250k. It's a risk on the order of riding a motorcycle from Oakland to San Jose in rush hour, I'd roughly estimate.
      - yardie 14y ago
        "I can totally reach into the back of this gigantic, flesh eating machine to move that widget a little bit to the right. Management might give me a raise for saving them money!" -famous last words of a former factory worker.
        rdl 14y ago
        "Do not look into laser with remaining eye" is the classic, though.
      - reeses 14y ago
        Enjoy that cancer from burning and/or airborne plastic/arsenic/mystery material in 20 years. And which part of Oakland? That can be a pretty broad range of risk. :-)
        rdl 14y ago
        "You wouldn't download a new lung, would you?" (RIAA ad in 2033). I lived in a tent 200m downwind of a 24x7 burning trash dump on a former Iraqi and then USAF military base, for some time, so I think I've got particulate risk checked already.
- protomyth 14y ago
  I was working at a place as a contractor and they had an amazing backup power system (expensive, diesel with batteries). Semi hits the power main outside the server room and for some reason the diesel never starts. Whole datacenter loses power in under 15 minutes. Always plan for the single event because no amount of money will keep a single site running.
  - tomjen3 14y ago
    The biggest danish ISP lost service for quite some time because the truck that were to deliver their new emergency power crashed into the mains transformer stations. Of course they are idiots, they should never have powered their own mains of before they got the new one installed. But it was still pretty funny.
- CamperBob2 14y ago
  Counterpoint: As a customer, I don't want to pay for the kind of redundancy that encourages employees to shut down the whole data center when a single tantalum cap pops in a power supply somewhere. The fact is, there aren't that many flammable things in a data center. Nobody is going to die because they wandered up and down the aisles after they "thought they smelled something burning," in the absence of any visible smoke. The guidelines in the highest-voted answer on the SO page make sense to me. 1: If you actually see smoke or fire in any significant amount, evacuate. 2: Make someone else aware of what's going on before doing anything else. 3: Keep your escape options open. 4: Think about how much time you can safely spend "guessing", and don't exceed it. 5: Don't second-guess your own common sense. You aren't paid to be a fireman or a hero.
- ghshephard 14y ago
  Almost nobody (for any reasonable definition of nobody) has immediate redundancy for their data center. I've worked for three $1B+ companies whose entire business was based on their data center being up and running - and none of them would have returned to service in less than 48 hours if a data center had gone down. "Not being prepared for them is unforgivable" - would mean that 99% of business do not deserve forgiveness. It just doesn't make sense to have that kind of redundancy for such a rare event for all but a very, very small minority of businesses. (Telecoms, Google, Stock Exchange, 911, etc...)
Aloisius 14y ago
I'm not sure immediately pressing the big red button because you mistook the smell a lead acid battery out gassing for something burning (with no smoke) is necessarily a measure of preparedness.

ck2 14y ago

Maybe we need a diy thermal sensor that plugs into an android or iphone device? Oh wow, it exists: http://www.instructables.com/id/Thermal-Imaging-Phone-Camera/ http://www.instructables.com/id/Thermal-Imaging-Phone-Camera... http://www.robhopeless.com/search/label/Thermal%20Imaging http://www.robhopeless.com/search/label/Thermal%20Imaging http://www.kickstarter.com/projects/andyrawson/ir-blue-thermal-imaging-smartphone-accessory http://www.kickstarter.com/projects/andyrawson/ir-blue-therm... http://rh-workshop-llc.myshopify.com/ http://rh-workshop-llc.myshopify.com/ Costs only $150 to make? Open-source: https://github.com/RHWorkshop/ https://github.com/RHWorkshop/

aw3c2 14y ago
Wow, I wonder what the range of that is. I.e. if I can use it to analyse the thermal loss of houses.
- ck2 14y ago
  I believe that's what the inventor originally made it for, if you look at his postings.
- walshemj 14y ago
  why do you need that you from my termofluids classes you should be able to calculate the heat loss for a given Delta T by taking a ball park figure for the materials ie so many square meters of brick Glass etc.
  - wiml 14y ago
    You use it when you don't know the material (because you're surveying an old house) or you don't know the delta-T (because you're looking for a hidden fire, grow op, etc).
    - NickNameNick 14y ago
      Or you're trying to track down head loss caused by construction flaws, especially leaks and drafts caused by cracks or gaps. Also leaks caused by improperly installed power points. Missing or settled insulation is also common.

hp50g 14y ago

Infra red camera!

rdl 14y ago

IR/thermal imaging cameras are SO USEFUL. I had a fire (bathroom fan caught fire due to being 45y old, knocked it down and extinguished it myself, but was worried about extension in the ceiling/duct). Oakland FD came out and used their IR camera to check the heat from the ceilings nearby. Hilariously they found a hot water pipe (running between bathroom and kitchen) and almost axed the ceiling open (turning $1500 in damage into $3k+), but their captain was smart and figured it out from another angle. Really tempted to hack an EOS 5Dm3 into an IR camera next. Not so much for fires as night vision, but it would be useful for fires too. I'm not sure how useful an IR camera is at detecting heat, since things which aren't yet on fire are not quite so infrared, though. I usually use a Fluke IR temp meter when cooking and to find hot wires/etc. in the datacenter, though.

Udo 14y ago
They are useful but they do have limitations. I'm not sure it would have necessarily detected the faulty battery in this case. Last year, there was a fire at my house and the FD searched for the source for three hours. With thermal imaging and everything. It was inside the walls, no open flame, just a lot of smoke and no clear readings on the imager. That was pretty frightening. (However when they finally did find it, they put it out in a couple of minutes.)
- rdl 14y ago
  I'm glad it motivated me to get ABC Dry and Halon 1211 extinguishers for both rooms and the car, at least. In a "real" datacenter, you should have smoke sensors which would map where heat/smoke is coming from (since you have controlled airflow, it should be obvious which rack or small group of racks was the source -- it doesn't just exhaust into the whole room). But it's pretty clear this wasn't a "real" datacenter by their lack of protocols for handling fire, it was some office server thing.
  - Udo 14y ago
    I'm just saying these things can quickly get more complicated than expected. I too had extinguishers handy at the time but of course I didn't know what to use them on (and neither did the FD for three hours). These electrical fires can be tricky to debug, especially when different kinds of barriers come into play. > But it's pretty clear this wasn't a "real" datacenter by their lack of protocols for handling fire, it was some office server thing. Probably. What's the right protocol though? In this case, it was apparently clear that something minor was amiss, nothing that would justify shutting down the whole thing. In any case, flooding the room with inert gas would probably not have made much of a difference, as it looks like the battery was never actually burning.

iSnow 14y ago

What kind of server room is this which is not equipped with smoke detectors?

dredmorbius 14y ago
Depending on how effective the ventilation system is / how diffuse the smoke is, the detectors may not do much to pinpoint where the smoke originated. Many DCs only have so many smoke detectors, at fairly wide intervals. There can be a lot of devices inside a 30' square.
gambiting 14y ago
If it's just the "melting plastic" kind of smoke, it won't trigger smoke detectors. And I believe that his battery wasn't actually on fire - if it was, then yes, smoke detectors would have triggered.
- HeyLaughingBoy 14y ago
  Oh yes, it will. I made my 12 year-old read the story (along with pictures) of a girl his age who was trapped in a burning house after he set off the smoke alarms at midnight by melting bits of plastic in his room. Hopefully he learned something that night.
  - eridius 14y ago
    How did setting off smoke detectors cause a fire?
    - HeyLaughingBoy 14y ago
      Poor sentence structure on my part. My son set off the smoke detectors by melting plastic in his bedroom with a cigarette lighter. To demonstrate the danger of what could come from this, I had him read a news report of a girl who was screwing around with fire and ended up being badly burned over most of her body.
14y ago

patio11 14y ago

You ask the fire marshal what was burning, about an hour or so after you've Big Red Button'ed the server room. This story is giving the Japanese engineer in me apoplexy.

elektronaut 14y ago
Acting quick and getting a fire under control often makes the difference between an emergency and a disaster. But only after you've made sure the building is being evacuated, and the fire department has been alerted. Don't be afraid to call the emergency number. They'll know what to do and walk you through it. Under no circumstance should you enter a room filled with smoke. Smoke inhalation is incredibly dangerous.
- MatthewPhillips 14y ago
  Smoke from electronics, especially.
  - kefka 14y ago
    Speaking of consumer electronics: When I went to the Midwest Reprap festival, for some reason power was being a bit flakey. The guy who's sitting across me had 2 printers using ATX power supplies. Suddenly, the PSU sparks out and then outgasses a stream of smoke pillar 4 foot in diameter for the next 2-3 minutes. Near the top of the building perhaps 3 stories up, it looks like a mini-Hiroshima with mushroom cloud going. And that was from a dinky ATX PSU.
    - GFischer 14y ago
      Heh, they once shipped 110v PSUs to us 220v users... I had to build systems at the time, the first one I plugged in made an awesome explosion :P
      - ordinary 14y ago
        My first (indeed, only) experience with the smell of burnt human flesh and hair was when a friend of mine flipped the switch on a PSU to 110v when it plugged in to 220v. He was fine, but the PSU didn't make it. RIP.

JimmaDaRustla 14y ago

Lead acid batteries will have a rotten smell after shorting out - more so in wet ones, but SLA will smell the same. If you wait to the point that an SLA smells, it has probably expanded and caused internal damage to your UPS/server rack, albeit minor if you can manage to get it out without dismantling anything.

jahabrewer 14y ago

It's been a while since I've been in IT, but don't failing UPS batteries put off fumes that destroy your lungs? A friend of a friend was a hero and shutdown his datacenter cleanly/recovered some hard drives during a situation like this. He got severe lung damage (not from fire).

whizzkid 14y ago

I couldn't answer the question there because of reputation thing. (just signed up) Here is a little bit different approach; If you are in the room and smelled the burn, that means something is already happened and you are dealing with its result and possible side effects, and that gives you possibly enough time before shutting down everything or getting out of the room. Your chance of not being harmed by this situation is high at least for 5-10 minutes. In this case, having a termal check would not help you a lot since burned hardware is most probably not functioning anymore and might be colder than the regular servers. The other option would be that it is still working but not causing any fire yet so heat is not much different than the usual. Now, smell is your only evidence, I am hoping you guys have air conditioner in the server room. Put it on the max level so that the smell will not be so strong everywhere. This can help you identify where the smell is coming from. Before you check the smell, get out of the server room, breath as much as fresh air you can, so that your realisation will be sharper when you get back to server room. Having your colleague with makes the process faster. This would be my first reaction to these kind of situations. It is of course costly to turn off whole system but don't forget that it is not important than you!

jeremyjh 14y ago

I'm not sure if everyone responding really read this all that carefully. There was absolutely no mention of smoke in this question. There was a "smell". If you drop an entire datacenter, you are easily looking at $100K+ in damages just to reset the room. So, getting a buddy and taking some time to look for the problem until it is found or until you actually are seeing smoke or other specific danger seems like a pretty reasonable course of action.

DanBC 14y ago
X times out of Y that's going to be a reasonable course of action. But one time someone will die, and at that point (because it's rare and we freak out at rare dangers) people will be up in arms about it, and about how stupid and irresponsible it is to not hit the button.
- ttrreeww 14y ago
  ...and that someone could be you.
- johngalt 14y ago
  Panic can be dangerous as well. A halon dump can asphyxiate someone who doesn't reach the oxygen mask or an exit in time. Dropping a whole server room without seeing any smoke or fire is silly. Do you pull a fire alarm if someone smokes a cigarette indoors?

blantonl 14y ago

It is pretty simple. You call the fire department. They have a TIC (thermal imaging camera) that can detect heat/overheating sources pretty quickly. Plus, it's kind of nice to have them on hand in case a smell progresses to a fire.

micro-ram 14y ago
I agree. Even if you call just to alert them that something is not right. Let them roll one truck just to have someone on hand in case someone gets burnt or shocked.
konstruktor 14y ago
I agree. People seem to imagine that all they do is put out fires with a hose. They are professionals who know how to manage potentially hazardous situations.

peterwwillis 14y ago

HVAC. Has no noticeable smoke (it'd probably be outside the building anyway) but pumps a burning smell into the room when the motors start dying or aren't oiled right. Don't hit the big red button just because you smell burning.

dattaway 14y ago

Lead based batteries often have a usable life of 3-5 years. Chances are, the others of that vintage are already dry and have already failed. Then they will rupture, often with smoke as their series connected brothers try to push electrons.

micro-ram 14y ago
I have seen plenty of UPS batteries swell up so big they can't be removed without disassembly. The only indication was the failed self test. OP did say it was the UPS in the rack next to his production DB. Don't forget Capacitor Plague. I still see it regularly. https://en.wikipedia.org/wiki/Capacitor_plague https://en.wikipedia.org/wiki/Capacitor_plague Have a plan, be safe.

protomyth 14y ago

I've had this happen to me once. No alerts and all boxes were up, but there was a smell in the room. I went machine by machine and UPS by UPS and nothing was wrong or burning(1). Next day we find out the breaker panel next door had a short that blew out several breakers. Smell was vented into the server room. So, not always your room, could be something else just as or more dangerous. 1) shut down all machines, unplug all UPSes, open every case

squozzer 14y ago

Temperature indicator stickers.

stevenrace 14y ago
At $5+/ea, that quickly adds up and difficult to parse in large numbers in duress. Whereas an IR camera is rather effective and perhaps cheaper.

darwinGod 14y ago

For some more context, this was the OP's previous question in Stackoverflow. http://serverfault.com/questions/420877/ive-inherited-a-rats-nest-of-cabling-what-now http://serverfault.com/questions/420877/ive-inherited-a-rats... Doesn't that change the entire question!

astrodust 14y ago
If there was a burning smell in that, yikes, hit the Big Red Button and turn in your resignation.
- micro-ram 14y ago
  Seeing that rats nest I would have let it burn!

FireBeyond 14y ago

If it's that important, buy a TIC (thermal imaging camera). They can be had for under $10,000 and will show you actual hotspots. Walk through, sweeping every item.

lobster45 14y ago

Thanks for this story. I am ordering a fire extinguisher for our server room now.

snarfy 14y ago

It helps to prioritize what you look for. 9 times out of 10 it's power related.

ChuckMcM 14y ago

I spent a couple of summers interning at IBM and one of the things they taught you in orientation was the sound of "imminent halon dump" (the alarm that said Halon was about to be used in the machine room). The instructions were, hold your breath and make for an exit immediately. Failure to do so would lead to asphyxiation.

johngalt 14y ago

'hmmm... probably UPS battery venting' click 'yup' When UPS batteries vent it has a distinctive odor. It's very pungent and sulfuric, but it doesn't smell like a fire or melting silicon. Any experienced operations guy has smelled it before. Additionally in most fire suppression systems the Big Red Button is the abort button. A well designed room will dump itself when it detects smoke after a short evacuation alarm. It's precisely designed to keep people from screwing around with a real fire. They must make the active decision to stop fire suppression rather than start fire suppression.

chris_wot 14y ago

Just an idea: the server room should be segmented into smaller areas with isolated power circuits. Using the sniff test, if you are truly concerned that you are about to have a fire, then it's only responsible to start an orderly powerdown to prevent equipment loss, and more importantly, prevent injuries to people. If you start shutting down areas of the datacentre that appear to be closest to the smoke, then you will have a better chance of locating the issue in the fastest possible timeframe, with minimum disruption. On top of this, if you then have critical infrastructure that you must keep running, then you keep your failover servers in different areas and failover to that equipment. I'm not a server or datacentre guy in any way, but doesn't this seem sensible?

Qantourisc 14y ago

Get in an electronics "expert" they know what component smells like what :)