4 ms·

It's research that's so valued they don't have a battery backup and don't page a grad student or two when the power goes off in the middle of the night. It's s

It's research that's so valued they don't have a battery backup and don't page a grad student or two when the power goes off in the middle of the night. It's so important they don't explain the problem to people who have access to the lab saying "hey. look at this sign over here."

hinata08 3y ago
Just watch a few CSB videos (among others) on YT to understand how some companies are run, when it comes to functional safety and ergonomics. This is unfortunately not taught everywhere apparently. Long story short, some companies would have done exactly the same management for something that would cause Armageddon. In the 80s in Paris, a train crashed when some train driver thought he was helpful by responding to an alarm and fixing the brakes himself instead of calling maintenance, and actually turned off the brakes because a poorly design switch made it look like the circuit was open. Plot twist, he was found to be not guilty of anything because company procedure were not clear. So a mere freezer probably went under the radar of management. In this case, a power loss could have been accidental as well. What if a fuse went out, or some unrelated piece if equipment caused an upstream GFCI to trip ? How can you trust a fridge and its power supply to protect 1M dollars for 20+ years, if 20 years worth of samples were destroyed? Power was just bound to be lost at some point in time, over a period of a few decades. (It even happens in datacenters) And some UPS would cost up to 2000$ ? Up to 500 more for the warning system to send a text ? IMO, functional safety should be a staple in university.
- jstarfish 3y ago
  > How can you trust a fridge and its power supply to protect 1M dollars for 20+ years, if 20 years worth of samples were destroyed? Power was just bound to be lost at some point in time, over a period of a few decades. (It even happens in datacenters) This very point makes the entire thing so ridiculous I'm not convinced this isn't the setup to outright fraud. Like a failing storeowner who starts making customers with children traverse the artisanal glasscraft section to get to the toy section. "Oopsies! You owe me $1M." I can't say this is about money (it's a university), but suspect maybe this project was doomed anyway and now its coordinator is staging sabotage to save face. The only thing that gives me pause is the thought that it's a little weird the cleaner was messing with the circuit breakers at all. The explanation offered makes little sense.
  - 542354234235 3y ago
    The cleaner cut the power at the circuit breaker. It isn’t like they had an extension cable running through the middle of the room that they tripped over. In what world is that equivalent to “making customers with children traverse the artisanal glasscraft section to get to the toy section”? The RPI campus does have UPS and emergency generators because it is important, so the whole lab is protected from power failures. They put a sign on the fridge for the short period of time that the alarm was going off while waiting for repair just in case someone wanted to do something crazy, like pull the fridge out and unplug it. You can’t account for every single thing someone might do. What if instead of going to the circuit breaker to cut the power, he opened the fridge and pulled all the samples out so he could clean inside? Or did just unplug it? or something else stupid? I’m pretty sure there are a lot of businesses that would lose a lot of money if staff members started flipping off random circuit breakers to various systems and pieces of equipment.
    - hinata08 3y ago
      > You can’t account for every single thing someone might do. yes you can. It's called security, and it has been a booming industry since at least 4 decades. People will do stupid things because they think differently, have different things in mind,... Never wonder "if" they will do it, but "when" > if staff members started flipping off random circuit breakers If they are accessible to them then they shouldn't cause damage. Just lock the cabinet if nobody should touch them. A lock is 30$, isn't it? A tag with a phone number for maintainance is 5$ ? > They put a sign on the fridge So they knew they were 1 switch away from disaster, human intervention or not. I hope it had proper SIL rating, if they are mad now it failed. Either humans are nice and well intentioned but stupid, and it's a shame to blame them when they just tried. Or they're really evil, and then you need to protect the system from them. Anyways, it's stupid to blame workers afterwards. UPS should have been directly between the fridge and the wall, without switch in the middle. And it should have warned staff. If not, they had accepted the risk. The fridge was also already malfunctioning and required maintainance (hence the temperature alarm) So I'm surprise there was no monitoring whatsoever, just a sign, and now they are mad about its failure. It's not consistent.
tekla 3y ago
Didn't read the article. They already had an alarm that was defective that tracked the fridge temp and would alarm when it went off. What would a battery backup would have done if the fridge itself is defective?
- rcme 3y ago
  Well in this case the fridge would have kept running after the power was cut at the breaker.
- smcg 3y ago
  Worth reading to answer these questions. Alarm was not defective. Repairs were delayed and this happened while waiting for a repair. It wasn't expected that samples would be damaged due to a slightly dropped temperature for 1 week. Most major flaw here was a lack of functional safety/training to avoid this sort of situation.
- ilyt 3y ago
  Then read the article because you wrote nonsense
justinclift 3y ago
> It's research that's so valued they don't have a battery backup ... Battery backup isn't going to help when someone turns the sample storage fridge off. :(
- lionkor 3y ago
  They turned off a circuit breaker, that sounds like there is definitely room for backup battery there.
- dingosity 3y ago
  That's actually exactly what battery backups do.
  - justinclift 3y ago
    You're saying that if you manually turn off $DEVICE, but have battery backup on $DEVICE, it'll actually keep on running until the battery runs out?
tdeck 3y ago
In case anyone wondered, it looks like you can buy a small ultra low temp freezer for about $10,000.
542354234235 3y ago
RPI does have emergency generators and UPS for their campus. The sign was on the fridge so that someone wouldn't do something crazy, like pull it away from the wall and unplug it. Someone going to the circuit breaker and killing the power to the whole area is not something any reasonable person would do or expect others to do, the same as I wouldn't expect the cleaners to "turn off the lights" by smashing the bulbs with a broom handle. It isn't exactly reasonable to come in after the fact and say "of course someone might try to turn off the lights by smashing the light bulbs. Obviously, you should have had cages around the lights to prevent this. How negligent of you."
- fatfingerd 3y ago
  Over 100 years I would expect 2 failures to overlap multiple times especially if you are willing to let one slide.. I'd have to assume they were getting paged when it failed and they failed to set a new bound to resume paging for a worse failure. Anything else could have been their second failure that caused an unacceptable temperature change instead of a cleaner misreading a breaker panel that apparently only needed one failure per 25 years. Not the most scientific cleaners meet not the sharpest planners.
itsoktocry 3y ago
All true, but also pretty bold of a cleaner to cut the power to silence an alarm in a laboratory...

hinata08 3y ago

Just watch a few CSB videos (among others) on YT to understand how some companies are run, when it comes to functional safety and ergonomics. This is unfortunately not taught everywhere apparently. Long story short, some companies would have done exactly the same management for something that would cause Armageddon. In the 80s in Paris, a train crashed when some train driver thought he was helpful by responding to an alarm and fixing the brakes himself instead of calling maintenance, and actually turned off the brakes because a poorly design switch made it look like the circuit was open. Plot twist, he was found to be not guilty of anything because company procedure were not clear. So a mere freezer probably went under the radar of management. In this case, a power loss could have been accidental as well. What if a fuse went out, or some unrelated piece if equipment caused an upstream GFCI to trip ? How can you trust a fridge and its power supply to protect 1M dollars for 20+ years, if 20 years worth of samples were destroyed? Power was just bound to be lost at some point in time, over a period of a few decades. (It even happens in datacenters) And some UPS would cost up to 2000$ ? Up to 500 more for the warning system to send a text ? IMO, functional safety should be a staple in university.

jstarfish 3y ago
> How can you trust a fridge and its power supply to protect 1M dollars for 20+ years, if 20 years worth of samples were destroyed? Power was just bound to be lost at some point in time, over a period of a few decades. (It even happens in datacenters) This very point makes the entire thing so ridiculous I'm not convinced this isn't the setup to outright fraud. Like a failing storeowner who starts making customers with children traverse the artisanal glasscraft section to get to the toy section. "Oopsies! You owe me $1M." I can't say this is about money (it's a university), but suspect maybe this project was doomed anyway and now its coordinator is staging sabotage to save face. The only thing that gives me pause is the thought that it's a little weird the cleaner was messing with the circuit breakers at all. The explanation offered makes little sense.
- 542354234235 3y ago
  The cleaner cut the power at the circuit breaker. It isn’t like they had an extension cable running through the middle of the room that they tripped over. In what world is that equivalent to “making customers with children traverse the artisanal glasscraft section to get to the toy section”? The RPI campus does have UPS and emergency generators because it is important, so the whole lab is protected from power failures. They put a sign on the fridge for the short period of time that the alarm was going off while waiting for repair just in case someone wanted to do something crazy, like pull the fridge out and unplug it. You can’t account for every single thing someone might do. What if instead of going to the circuit breaker to cut the power, he opened the fridge and pulled all the samples out so he could clean inside? Or did just unplug it? or something else stupid? I’m pretty sure there are a lot of businesses that would lose a lot of money if staff members started flipping off random circuit breakers to various systems and pieces of equipment.
  - hinata08 3y ago
    > You can’t account for every single thing someone might do. yes you can. It's called security, and it has been a booming industry since at least 4 decades. People will do stupid things because they think differently, have different things in mind,... Never wonder "if" they will do it, but "when" > if staff members started flipping off random circuit breakers If they are accessible to them then they shouldn't cause damage. Just lock the cabinet if nobody should touch them. A lock is 30$, isn't it? A tag with a phone number for maintainance is 5$ ? > They put a sign on the fridge So they knew they were 1 switch away from disaster, human intervention or not. I hope it had proper SIL rating, if they are mad now it failed. Either humans are nice and well intentioned but stupid, and it's a shame to blame them when they just tried. Or they're really evil, and then you need to protect the system from them. Anyways, it's stupid to blame workers afterwards. UPS should have been directly between the fridge and the wall, without switch in the middle. And it should have warned staff. If not, they had accepted the risk. The fridge was also already malfunctioning and required maintainance (hence the temperature alarm) So I'm surprise there was no monitoring whatsoever, just a sign, and now they are mad about its failure. It's not consistent.

tekla 3y ago

Didn't read the article. They already had an alarm that was defective that tracked the fridge temp and would alarm when it went off. What would a battery backup would have done if the fridge itself is defective?

rcme 3y ago
Well in this case the fridge would have kept running after the power was cut at the breaker.
smcg 3y ago
Worth reading to answer these questions. Alarm was not defective. Repairs were delayed and this happened while waiting for a repair. It wasn't expected that samples would be damaged due to a slightly dropped temperature for 1 week. Most major flaw here was a lack of functional safety/training to avoid this sort of situation.
ilyt 3y ago
Then read the article because you wrote nonsense

justinclift 3y ago

> It's research that's so valued they don't have a battery backup ... Battery backup isn't going to help when someone turns the sample storage fridge off. :(

lionkor 3y ago
They turned off a circuit breaker, that sounds like there is definitely room for backup battery there.
dingosity 3y ago
That's actually exactly what battery backups do.
- justinclift 3y ago
  You're saying that if you manually turn off $DEVICE, but have battery backup on $DEVICE, it'll actually keep on running until the battery runs out?

tdeck 3y ago

In case anyone wondered, it looks like you can buy a small ultra low temp freezer for about $10,000.

542354234235 3y ago

RPI does have emergency generators and UPS for their campus. The sign was on the fridge so that someone wouldn't do something crazy, like pull it away from the wall and unplug it. Someone going to the circuit breaker and killing the power to the whole area is not something any reasonable person would do or expect others to do, the same as I wouldn't expect the cleaners to "turn off the lights" by smashing the bulbs with a broom handle. It isn't exactly reasonable to come in after the fact and say "of course someone might try to turn off the lights by smashing the light bulbs. Obviously, you should have had cages around the lights to prevent this. How negligent of you."

fatfingerd 3y ago
Over 100 years I would expect 2 failures to overlap multiple times especially if you are willing to let one slide.. I'd have to assume they were getting paged when it failed and they failed to set a new bound to resume paging for a worse failure. Anything else could have been their second failure that caused an unacceptable temperature change instead of a cleaner misreading a breaker panel that apparently only needed one failure per 25 years. Not the most scientific cleaners meet not the sharpest planners.

itsoktocry 3y ago

All true, but also pretty bold of a cleaner to cut the power to silence an alarm in a laboratory...