4 ms·

I think they're just large, complicated physical things where stuff goes wrong, even if you're maintaining and testing. My best story was as a young student tu

I think they're just large, complicated physical things where stuff goes wrong, even if you're maintaining and testing. My best story was as a young student turning up for my helpdesk shift at about 5:50am to a phalanx of fire engines and the Hazmat team. The generator in the basement was maintained and tested. This time it had started when power went out, but there was a pump that filled a holding tank from a 10,000ish litre tank under the building (speced to run for a week apparently). A pipe had cracked, and so it just kept pumping fuel on to the floor, carpark and into the local stream ... The only person in the building overnight was the operator (tape changer and report runner) some 5 stories above all this, who eventually smelt it and eventually investgated. It was a massive nightmare, and the place stank for weeks. Then there was the time the halon system went off unnecessarily...

_nalply 9y ago
I have a similar story of a smallish colocation site located in the cellar of a high school building in Switzerland where we hosted two servers for a small company ten years ago. The operator company had a generator at the site. During a thunderstorm there was a power outage, so at first all was fine. However a lightning nearby destroyed some critical circuitry of the generator. I additionally made the mistake of putting the watchdog system in the same colocation so that I didn't even get a notification about the outage. But after all I would have been powerless anyway.
robryk 9y ago
The particular failure mode you're talking about is reasonably easy to reliably remove: all fuel piping should be under lower than atmospheric pressure. You accomplish this by installing suction pumps next to consumers (and install additional pumps if the height difference is too large). That way, if a pipe breaks (except for the short pieces of pipe between the pipe and consumer), there will be no sustained fuel leak.
- Johnny555 9y ago
  Building constraints don't always make it that easy -- the generator could be below the tank level, so siphon action can still siphon the tank empty, or it can be a few floors about the tank, too high to suck fuel from the tank so it has to be pushed up from below.
  - robryk 9y ago
    > the generator could be below the tank level This is what anti-siphon valves are used to prevent: they are valves that open only when sufficient suction is present in the fuel line leaving the tank. Thus, when the line breaks or the pump fails, the valve will close. > or it can be a few floors about the tank, too high to suck fuel from the tank so it has to be pushed up from below. True, it has to be pushed up then. Alas, you can have a sequence of suction pumps. Note that otherwise you'd need to have high-pressure fuel piping, which might cause more problems than the possibility of feeding a fire indefinitely.
- robocat 9y ago
  So now the system fails due to frothy diesel. Or due to failures of the additional pumps (adding mechanical complexity to a system often decreases overall reliability)
  - robryk 9y ago
    Yes. This is a tradeoff between a chance of catastrophic failure (fire fed by fuel from the storage tank) and "normal" failure.

_nalply 9y ago

I have a similar story of a smallish colocation site located in the cellar of a high school building in Switzerland where we hosted two servers for a small company ten years ago. The operator company had a generator at the site. During a thunderstorm there was a power outage, so at first all was fine. However a lightning nearby destroyed some critical circuitry of the generator. I additionally made the mistake of putting the watchdog system in the same colocation so that I didn't even get a notification about the outage. But after all I would have been powerless anyway.

robryk 9y ago

The particular failure mode you're talking about is reasonably easy to reliably remove: all fuel piping should be under lower than atmospheric pressure. You accomplish this by installing suction pumps next to consumers (and install additional pumps if the height difference is too large). That way, if a pipe breaks (except for the short pieces of pipe between the pipe and consumer), there will be no sustained fuel leak.

Johnny555 9y ago
Building constraints don't always make it that easy -- the generator could be below the tank level, so siphon action can still siphon the tank empty, or it can be a few floors about the tank, too high to suck fuel from the tank so it has to be pushed up from below.
- robryk 9y ago
  > the generator could be below the tank level This is what anti-siphon valves are used to prevent: they are valves that open only when sufficient suction is present in the fuel line leaving the tank. Thus, when the line breaks or the pump fails, the valve will close. > or it can be a few floors about the tank, too high to suck fuel from the tank so it has to be pushed up from below. True, it has to be pushed up then. Alas, you can have a sequence of suction pumps. Note that otherwise you'd need to have high-pressure fuel piping, which might cause more problems than the possibility of feeding a fire indefinitely.
robocat 9y ago
So now the system fails due to frothy diesel. Or due to failures of the additional pumps (adding mechanical complexity to a system often decreases overall reliability)
- robryk 9y ago
  Yes. This is a tradeoff between a chance of catastrophic failure (fire fed by fuel from the storage tank) and "normal" failure.