7 ms·

Wishful thinking, but I hope an engineer somewhere got to ram a door down to fix a global outage. For the stories.

bluGill 1y ago
I assume they needed their own air supply because the automatic poison gas system was activating. Then they had to dodge lazers to get to the one button that would stop the nuclear missle launch. add a bunch of other poinless scifi and evil villan lair tropes in as well...
- donalhunt 1y ago
  Most datacenters are fairly boring to be honest. The most exciting thing likely to happen is some sheet metal ripping your hand open because you didn't wear gloves. Still have my "my other datacenter is made of razorblades and hate" sticker. \o/
  - formerly_proven 1y ago
    They do commonly have poisonous gas though.
    - maaaaattttt 1y ago
      Not sure if you’re joking but a relatively small datacenter I’m familiar with has reduced oxygen in it to prevent fires. If you were to break in unannounced you would faint or maybe worse (?).
      - bob778 1y ago
        Not quite - while you can reduce oxygen levels, they have to be kept within 4pp so at worst, will make you light headed. Many athletes train at the same levels though so it’s easy to overcome.
        waste_monk 1y ago
        That'd make for a decent heist comedy - a bunch of former professional athletes get hired to break in to a low-oxygen data center, but the plan goes wrong and they have to use their sports skills in improbable ways to pull it off.
    - ArnoVW 1y ago
      And lasers come to think of it
    - UltraSane 1y ago
      No FM200 isn't poisonous.
      - bdangubic 1y ago
        tell that to my dead uncle jack :)
    - mrgoldenbrown 1y ago
      Halon was used back in the day for fire suppression but I thought it was only dangerous at high enough concentrations to suffocate you by displacing oxygen.
      - userbinator 1y ago
        To be of any use, it also has to suffocate a fire.
        oasisbob 1y ago
        Halon doesn't work that way, by displacing oxygen. Flame chemistry is weird. Halogenated fire suppression agents work by making Hydrogen (!) out of free radicals. https://www.nist.gov/system/files/documents/el/fire_research/Fleming-Chemical-Fire-Suppressants-How-can-we-Replace-Halon.pdf https://www.nist.gov/system/files/documents/el/fire_research...
    - christkv 1y ago
      I had a summer job at a hospital one year in the data center when an electrician managed to trigger the halon system and we all had to evacuate and wait for the process to finish and the gas to vent. The four firetrucks and station master who shoved up was both annoyed and relieved it was not real.
  - tacticus 1y ago
    there are datacentres not made of razorblades and hate?
jedberg 1y ago
Way back when I worked at eBay, we once had a major outage and needed datacenter access. The datacenter process normally took about 5 minutes per person to verify identity and employment, and then scan past the biometric scanners. On that day, the VP showed up and told the security staff, "just open all the doors!". So they did. If you knew where the datacenter was, you could just walk-in in mess with eBay servers. But since we were still a small ops team, we pretty much knew everyone who was supposed to be there. So security was basically "does someone else recognize you?".
- E39M5S62 1y ago
  That sounds like an Equinix datacenter. They were painfully slow at 350 E. Cermak.
  - jedberg 1y ago
    It wasn't Equinix, but I think the vendor was acquired by them. I don't actually blame them, I appreciated their security procedures. The five minutes usually didn't matter.
- chasd00 1y ago
  way back when DC's were secure but not _that secure_ i social engineered my way close enough to our rack without ID to hit a reset button before getting thrown out. /those were the days
  - jedberg 1y ago
    Just to test the security, or...?
    - chasd00 1y ago
      late reply but, no, i really needed to hit the button but didn't have valid ID at the time. My driver's license was expired and i couldn't get it renewed because of a outstanding tickets iirc. I was able to talk my way in and had been there many times before so knew my way around and what words to say. I was able to do what i needed before another admin came up and told me that without valid ID they have no choice but to ask me to leave (probably like an insurance thing). I was being a bit dramatic when i said "getting thrown out" the datacenter guys were very nice and almost apologetic about asking me to leave.
wolpoli 1y ago
The story was that they had to use an angle grinder to get in.
- bombcar 1y ago
  I prefer to use a sawzall and just go through the wall.
  - adrianmonk 1y ago
    The memory is hazy since it was 15+ years ago, but I'm fairly sure I knew someone who worked at a company whose servers were stolen this way. The thieves had access to the office building but not the server room. They realized the server room shared a wall with a room that they did have access to, so they just used a sawzall to make an additional entrance.
    - chasd00 1y ago
      my across the street neighbor had some expensive bikes stolen this way. The thieves just cut a hole in the side of their garage from the alley, security cameras were facing the driveway and with nothing on the alley side. We (the neighborhood) think they were targeted specifically for the bikes as nothing else was stolen and your average crack head isn't going to make that level of effort.
  - oblio 1y ago
    That would be a sawswall, in that case.
- jonbiggums22 1y ago
  I remember hearing Google early in it's history had some sort of emergency back up codes that they encased in concrete to prevent them becoming a casual part of the process and they needed a jack hammer and a couple hours when the supposedly impossible happened after only a couple years.
  - 1y ago
lazide 1y ago
Not an active datacenter, but I did get to use a fire extinguisher to knock out a metal-mesh-reinforced window in a secure building once because no one knew where the keys were for an important room. Management was not happy, but I didn’t get in trouble for it. And yes, it was awesome. Surprisingly easy, especially since the fire extinguisher was literally right next to it.
- geephroh 1y ago
  Sometimes a little good old fashioned mayhem is good for employee morale
  - lazide 1y ago
    Every good firefighter knows this feeling. Nothing says ‘go ahead, destroy that shit’ like money going up in smoke if you don’t. P.S. don’t park in front of fire hydrants, because they will have a shit eating grin on their face when they destroy your car- ahem - clear the obstacle - when they need to use it to stop a fire.

I assume they needed their own air supply because the automatic poison gas system was activating. Then they had to dodge lazers to get to the one button that would stop the nuclear missle launch. add a bunch of other poinless scifi and evil villan lair tropes in as well...

donalhunt 1y ago
Most datacenters are fairly boring to be honest. The most exciting thing likely to happen is some sheet metal ripping your hand open because you didn't wear gloves. Still have my "my other datacenter is made of razorblades and hate" sticker. \o/
- formerly_proven 1y ago
  They do commonly have poisonous gas though.
  - maaaaattttt 1y ago
    Not sure if you’re joking but a relatively small datacenter I’m familiar with has reduced oxygen in it to prevent fires. If you were to break in unannounced you would faint or maybe worse (?).
    - bob778 1y ago
      Not quite - while you can reduce oxygen levels, they have to be kept within 4pp so at worst, will make you light headed. Many athletes train at the same levels though so it’s easy to overcome.
      - waste_monk 1y ago
        That'd make for a decent heist comedy - a bunch of former professional athletes get hired to break in to a low-oxygen data center, but the plan goes wrong and they have to use their sports skills in improbable ways to pull it off.
  - ArnoVW 1y ago
    And lasers come to think of it
  - UltraSane 1y ago
    No FM200 isn't poisonous.
    - bdangubic 1y ago
      tell that to my dead uncle jack :)
  - mrgoldenbrown 1y ago
    Halon was used back in the day for fire suppression but I thought it was only dangerous at high enough concentrations to suffocate you by displacing oxygen.
    - userbinator 1y ago
      To be of any use, it also has to suffocate a fire.
      - oasisbob 1y ago
        Halon doesn't work that way, by displacing oxygen. Flame chemistry is weird. Halogenated fire suppression agents work by making Hydrogen (!) out of free radicals. https://www.nist.gov/system/files/documents/el/fire_research/Fleming-Chemical-Fire-Suppressants-How-can-we-Replace-Halon.pdf https://www.nist.gov/system/files/documents/el/fire_research...
  - christkv 1y ago
    I had a summer job at a hospital one year in the data center when an electrician managed to trigger the halon system and we all had to evacuate and wait for the process to finish and the gas to vent. The four firetrucks and station master who shoved up was both annoyed and relieved it was not real.
- tacticus 1y ago
  there are datacentres not made of razorblades and hate?

jedberg 1y ago

Way back when I worked at eBay, we once had a major outage and needed datacenter access. The datacenter process normally took about 5 minutes per person to verify identity and employment, and then scan past the biometric scanners. On that day, the VP showed up and told the security staff, "just open all the doors!". So they did. If you knew where the datacenter was, you could just walk-in in mess with eBay servers. But since we were still a small ops team, we pretty much knew everyone who was supposed to be there. So security was basically "does someone else recognize you?".

E39M5S62 1y ago
That sounds like an Equinix datacenter. They were painfully slow at 350 E. Cermak.
- jedberg 1y ago
  It wasn't Equinix, but I think the vendor was acquired by them. I don't actually blame them, I appreciated their security procedures. The five minutes usually didn't matter.
chasd00 1y ago
way back when DC's were secure but not _that secure_ i social engineered my way close enough to our rack without ID to hit a reset button before getting thrown out. /those were the days
- jedberg 1y ago
  Just to test the security, or...?
  - chasd00 1y ago
    late reply but, no, i really needed to hit the button but didn't have valid ID at the time. My driver's license was expired and i couldn't get it renewed because of a outstanding tickets iirc. I was able to talk my way in and had been there many times before so knew my way around and what words to say. I was able to do what i needed before another admin came up and told me that without valid ID they have no choice but to ask me to leave (probably like an insurance thing). I was being a bit dramatic when i said "getting thrown out" the datacenter guys were very nice and almost apologetic about asking me to leave.

wolpoli 1y ago

The story was that they had to use an angle grinder to get in.

bombcar 1y ago
I prefer to use a sawzall and just go through the wall.
- adrianmonk 1y ago
  The memory is hazy since it was 15+ years ago, but I'm fairly sure I knew someone who worked at a company whose servers were stolen this way. The thieves had access to the office building but not the server room. They realized the server room shared a wall with a room that they did have access to, so they just used a sawzall to make an additional entrance.
  - chasd00 1y ago
    my across the street neighbor had some expensive bikes stolen this way. The thieves just cut a hole in the side of their garage from the alley, security cameras were facing the driveway and with nothing on the alley side. We (the neighborhood) think they were targeted specifically for the bikes as nothing else was stolen and your average crack head isn't going to make that level of effort.
- oblio 1y ago
  That would be a sawswall, in that case.
jonbiggums22 1y ago
I remember hearing Google early in it's history had some sort of emergency back up codes that they encased in concrete to prevent them becoming a casual part of the process and they needed a jack hammer and a couple hours when the supposedly impossible happened after only a couple years.
- 1y ago

lazide 1y ago

Not an active datacenter, but I did get to use a fire extinguisher to knock out a metal-mesh-reinforced window in a secure building once because no one knew where the keys were for an important room. Management was not happy, but I didn’t get in trouble for it. And yes, it was awesome. Surprisingly easy, especially since the fire extinguisher was literally right next to it.

geephroh 1y ago
Sometimes a little good old fashioned mayhem is good for employee morale
- lazide 1y ago
  Every good firefighter knows this feeling. Nothing says ‘go ahead, destroy that shit’ like money going up in smoke if you don’t. P.S. don’t park in front of fire hydrants, because they will have a shit eating grin on their face when they destroy your car- ahem - clear the obstacle - when they need to use it to stop a fire.