4 ms·

I've never worked in a data center that did cooling and power correctly. Everyone thinks they're doing it right, and then street power gets cut - there's signif

I've never worked in a data center that did cooling and power correctly. Everyone thinks they're doing it right, and then street power gets cut - there's significant impact, ops teams scramble to contain, and finally there's the finger-pointing.

w0m 2y ago
I mean; it's impossible to plan for everything, and I'd argue that if you actually did plan for everything; it would be so extraordinarily overbuilt that it couldn't be considered 'correct'.
danudey 2y ago
> then street power gets cut Or the electrician doing maintenance on the backup generator doesn't properly connect the bypass and no one notices until he disconnects the generator and the entire DC instantly goes quiet. Or your DC provisions rack space without knowing which servers are redundant with which other servers, and suddenly when two services go from 10% CPU use to 100% CPU across ten servers the breaker for that circuit gives up entirely and takes down your entire business.
doubled112 2y ago
The colo I’m used to has survived multiple switch overs to backup and then to diesel generators without a blip that I could detect. I say “I’m used to” because having things there has spanned more than one job. One power outage was days to a week. Don’t recall exactly. It’s possible to do it right.
- SoftTalker 2y ago
  Yes it's possible. But it's not cheap. If you buy a bunch of UPS and a few generators (you need more than one, in case it doesn't start) and don't maintain them regularly and test them regularly that's when you get some bad surprises.
jms 2y ago
The first time we tested cutting the power back in the day, the backup generator didn't fire! Turns out someone had pushed the big red stop button, which remains pushed in until reset. That would have been a major problem if we'd had a nighttime power outage. After that we ran regular switchover testing :) The other time we ran into trouble was after someone drove a car into the local power substation. Our systems all ran fine for the immediate outage, but the power company's short term fix was to re-route power, which caused our voltage to be low enough for our UPS batteries to slowly drain without tripping over to the generator. That was a week or two of manually pumping diesel into the generator tank so we could keep the UPS batteries topped up.
roydivision 2y ago
I have, and least power. I worked in a DC with 4 very large diesel generators, each backed up by a multi-ton flywheel that managed the transition between a power cut and the generators taking over. Area wide power cut, winter afternoon so it was already getting dark. The two signs I knew there was something wrong were that all the lights went out outside, ie other businesses, street lighting etc. And my internet connection stopped working. Nothing else in the DC was affected. Even the elevator was working.
- tomcam 2y ago
  Amazing story, and now the flywheel is living rent free in a steampunk-inspired corner of my brain. What are these called so I can look them up on the net? Like this maybe? https://www.torus.co/torus-flywheel https://www.torus.co/torus-flywheel
  - viraptor 2y ago
    They're chonky devices which were not really off-the-shelf until the last decade, as far as I know. There's very few images of them, but a smaller one looks like this: https://www.pv-magazine.com/2018/03/14/pilot-project-for-flywheel-storage-underway-in-hawaii/ https://www.pv-magazine.com/2018/03/14/pilot-project-for-fly...
    - tomcam 2y ago
      Very cool. Somehow I missed the idea of flywheels used to store energy. I assumed one that small would peter out in seconds.
      - roydivision 2y ago
        These were huge. In fact iirc the power that came into the building spun them, and the DC ran off the resultant generated electricity. So the risk at cutover is minimal, there is in fact no loss of electricity unless the wheel drops below the required revs. One of these units blew at one point. We had 4 and only needed two running, so no big deal. The company who managed the whole thing (Swiss) came to replace it. Amazing job, they had to put it on small rollers, like industrial roller skates, then embed hooks in the walls at each corridor junction, and slowly winch the thing along, it was like watching the minute hand of a clock. Then the whole process in reverse to bring in the new one. Was fascinating to watch. The guy in charge was a giant, built like a brick outhouse. They knew their stuff.

w0m 2y ago

I mean; it's impossible to plan for everything, and I'd argue that if you actually did plan for everything; it would be so extraordinarily overbuilt that it couldn't be considered 'correct'.

danudey 2y ago

> then street power gets cut Or the electrician doing maintenance on the backup generator doesn't properly connect the bypass and no one notices until he disconnects the generator and the entire DC instantly goes quiet. Or your DC provisions rack space without knowing which servers are redundant with which other servers, and suddenly when two services go from 10% CPU use to 100% CPU across ten servers the breaker for that circuit gives up entirely and takes down your entire business.

doubled112 2y ago

The colo I’m used to has survived multiple switch overs to backup and then to diesel generators without a blip that I could detect. I say “I’m used to” because having things there has spanned more than one job. One power outage was days to a week. Don’t recall exactly. It’s possible to do it right.

SoftTalker 2y ago
Yes it's possible. But it's not cheap. If you buy a bunch of UPS and a few generators (you need more than one, in case it doesn't start) and don't maintain them regularly and test them regularly that's when you get some bad surprises.

jms 2y ago

The first time we tested cutting the power back in the day, the backup generator didn't fire! Turns out someone had pushed the big red stop button, which remains pushed in until reset. That would have been a major problem if we'd had a nighttime power outage. After that we ran regular switchover testing :) The other time we ran into trouble was after someone drove a car into the local power substation. Our systems all ran fine for the immediate outage, but the power company's short term fix was to re-route power, which caused our voltage to be low enough for our UPS batteries to slowly drain without tripping over to the generator. That was a week or two of manually pumping diesel into the generator tank so we could keep the UPS batteries topped up.

roydivision 2y ago

I have, and least power. I worked in a DC with 4 very large diesel generators, each backed up by a multi-ton flywheel that managed the transition between a power cut and the generators taking over. Area wide power cut, winter afternoon so it was already getting dark. The two signs I knew there was something wrong were that all the lights went out outside, ie other businesses, street lighting etc. And my internet connection stopped working. Nothing else in the DC was affected. Even the elevator was working.

tomcam 2y ago
Amazing story, and now the flywheel is living rent free in a steampunk-inspired corner of my brain. What are these called so I can look them up on the net? Like this maybe? https://www.torus.co/torus-flywheel https://www.torus.co/torus-flywheel
- viraptor 2y ago
  They're chonky devices which were not really off-the-shelf until the last decade, as far as I know. There's very few images of them, but a smaller one looks like this: https://www.pv-magazine.com/2018/03/14/pilot-project-for-flywheel-storage-underway-in-hawaii/ https://www.pv-magazine.com/2018/03/14/pilot-project-for-fly...
  - tomcam 2y ago
    Very cool. Somehow I missed the idea of flywheels used to store energy. I assumed one that small would peter out in seconds.
    - roydivision 2y ago
      These were huge. In fact iirc the power that came into the building spun them, and the DC ran off the resultant generated electricity. So the risk at cutover is minimal, there is in fact no loss of electricity unless the wheel drops below the required revs. One of these units blew at one point. We had 4 and only needed two running, so no big deal. The company who managed the whole thing (Swiss) came to replace it. Amazing job, they had to put it on small rollers, like industrial roller skates, then embed hooks in the walls at each corridor junction, and slowly winch the thing along, it was like watching the minute hand of a clock. Then the whole process in reverse to bring in the new one. Was fascinating to watch. The guy in charge was a giant, built like a brick outhouse. They knew their stuff.