8 ms·

RPI sues after cleaner's error allegedly destroyed decades of research

36 points by blindriver 3y ago

bestcoder69 3y ago
Did they keep all their data in RAM on a single server too?
intrasight 3y ago
Can't help but think RPI shares some culpability here if they allowed anyone to throw that switch. Surely electrical codes allow for the protection of such breaker switches.
- intrasight 3y ago
  Dating myself here, but reminds me of the episode of the '70s TV program "Salvage 1" about a scrap dealer (played by Andy Griffith) who builds a rocket from parts he finds. On their maiden voyage (I think) the cleaning guy unplugs the guidance computer so that he can plug in his vacuum cleaner.
  - aaron695 3y ago
    [dead]
lowbloodsugar 3y ago
RPI clearly had the information and obligation to understand the nature of their problem and to implement an effective solution. Turns out they were unable to do so. One might say that the scientists were not trained on how to implement effective operational procedures, believing instead that a piece of paper with a message on it would be effective. Their claim that a cleaning company should have been able to anticipate their failure and do a better job at operational assessment than RPI seems like a losing argument.
rcme 3y ago
Apparently they had a sign that explained how to mute the alarm. I think that’s going to work against them. They were aware that the alarm could trigger erroneously and didn’t fix the issue.
- tekla 3y ago
  Didn't read the article. The article clearly states the fridge was scheduled for repair, but due to COVID was delayed.
  - rcme 3y ago
    Why does that matter? They knew it was broken but didn’t explicitly notify the cleaning company to avoid the fridge. Like what is the argument exactly? “I knew this was broken and didn’t tell you about it. Now you owe me money because the broken thing interfered with your work.”
    - justinclift 3y ago
      The sign was specifically for the cleaner, explaining the problem, plus instructing them not to clean the area. It also told them how to mute the alarm sound if it was annoying. In hind sight it would have been better to pre-contact the cleaning company and make sure the cleaner in question understood the situation. But that's only from knowing after the fact that the cleaner was hmmm... a friggin idiot. ;)
      - rcme 3y ago
        Do you really think the cleaner looks at every thing posted in the lab as he cleans? Very unlikely. Agreements require a meeting of the minds. The only idiots I see are the ones that lost decades of research.
        justinclift 3y ago
        Depends on the sign. I guess I'm assuming it'd be "something obvious" but that could indeed be a really dumb assumption on my part. :)
dingosity 3y ago
It's research that's so valued they don't have a battery backup and don't page a grad student or two when the power goes off in the middle of the night. It's so important they don't explain the problem to people who have access to the lab saying "hey. look at this sign over here."
- hinata08 3y ago
  Just watch a few CSB videos (among others) on YT to understand how some companies are run, when it comes to functional safety and ergonomics. This is unfortunately not taught everywhere apparently. Long story short, some companies would have done exactly the same management for something that would cause Armageddon. In the 80s in Paris, a train crashed when some train driver thought he was helpful by responding to an alarm and fixing the brakes himself instead of calling maintenance, and actually turned off the brakes because a poorly design switch made it look like the circuit was open. Plot twist, he was found to be not guilty of anything because company procedure were not clear. So a mere freezer probably went under the radar of management. In this case, a power loss could have been accidental as well. What if a fuse went out, or some unrelated piece if equipment caused an upstream GFCI to trip ? How can you trust a fridge and its power supply to protect 1M dollars for 20+ years, if 20 years worth of samples were destroyed? Power was just bound to be lost at some point in time, over a period of a few decades. (It even happens in datacenters) And some UPS would cost up to 2000$ ? Up to 500 more for the warning system to send a text ? IMO, functional safety should be a staple in university.
  - jstarfish 3y ago
    > How can you trust a fridge and its power supply to protect 1M dollars for 20+ years, if 20 years worth of samples were destroyed? Power was just bound to be lost at some point in time, over a period of a few decades. (It even happens in datacenters) This very point makes the entire thing so ridiculous I'm not convinced this isn't the setup to outright fraud. Like a failing storeowner who starts making customers with children traverse the artisanal glasscraft section to get to the toy section. "Oopsies! You owe me $1M." I can't say this is about money (it's a university), but suspect maybe this project was doomed anyway and now its coordinator is staging sabotage to save face. The only thing that gives me pause is the thought that it's a little weird the cleaner was messing with the circuit breakers at all. The explanation offered makes little sense.
tonymillion 3y ago
So they keep millions of dollars worth of samples in a single freezer in a single location? The loss of the samples was an inevitability and regardless of who pulled the plug - a cleaner, a power cut or a faulty compressor they are solely responsible for the loss. Assuming a cleaner would read a note attached to a freezer, given they no doubt have to see lab notes all over the place - and probably have instructions not to move or read the notes - is completely delusional. IMO the blame falls squarely at the feet of the researchers and RPI and at worst the cleaning company should agree to terminate the contract (thus losing the $1.4 million per year deal they had).
- smcg 3y ago
  I mean, looking at a site now that sells cryogenic freezers, only one model goes colder than -86C, and this was going down to less than -110C. this is expensive and specialized equipment. I understand not having a second one but they needed more safety and redundancy on this one.
  - tonymillion 3y ago
    The article mentions Fahrenheit (-112f), which means it was -80C, so the first one you found would have been "good enough". A quick google finds a -80C freezer for $14000 [0] a fraction of the $1million they've "lost". [0] [Fisherbrand™ Isotemp™ Ultra Low Temperature Freezer](https://www.fishersci.com/shop/products/isotemp-ultra-low-temperature-freezer-25/IUE30086FA# https://www.fishersci.com/shop/products/isotemp-ultra-low-te...)
    - scrps 3y ago
      While I agree RPI screwed the pooch on this one; I am surprised for $14k the damn power switch isnt under a locked panel/switch cover given what it is used for.
JoeAltmaier 3y ago
Reminds me of the company that sequestered private keys. Some idiot VP went to the vault, took out the master list and email'd it to a friend who had forgot their key. Gutting the entire value of the company and ruining it forever. People do the damndest things. Critical stuff needs a little extra care put in place or some fool will inevitably do the thing-that-must-not-be-done. Sometimes I have no confidence that a mature civilization is even possible if it's composed of human beings.
- ilyt 3y ago
  > Sometimes I have no confidence that a mature civilization is even possible if it's composed of human beings. Of course it is, just takes some lawyers and PR to change the meaning of "mature"
- vinyl7 3y ago
  > Sometimes I have no confidence that a mature civilization is even possible if it's composed of human beings. It boggles my mind that humans have survived for so long considering how suicidal and self destructive we are as a species
  - kjs3 3y ago
    For every one of us monkeys who says "hey, y'all...watch this!" and does something insanely stupid, there are always at least a couple who take a step back, make sure we're out of the blast radius, and think "this guy is going to make a great warning to the others"...thus preserving the species for another day.
- tangjurine 3y ago
  What company?
DoItToMe81 3y ago
If he was willing to risk all this, imagine just how fucking annoying that alarm must have been.
tangjurine 3y ago
The guy said he was trying to be helpful after hearing the annoying alarm by turning the electricity on. Either he's really dumb, or he just doesn't care about the consequences and then makes up some bs excuse after. My money is on the latter.

36 points by blindriver 3y ago

Did they keep all their data in RAM on a single server too?

Can't help but think RPI shares some culpability here if they allowed anyone to throw that switch. Surely electrical codes allow for the protection of such breaker switches.

intrasight 3y ago
Dating myself here, but reminds me of the episode of the '70s TV program "Salvage 1" about a scrap dealer (played by Andy Griffith) who builds a rocket from parts he finds. On their maiden voyage (I think) the cleaning guy unplugs the guidance computer so that he can plug in his vacuum cleaner.
- aaron695 3y ago
  [dead]

lowbloodsugar 3y ago

RPI clearly had the information and obligation to understand the nature of their problem and to implement an effective solution. Turns out they were unable to do so. One might say that the scientists were not trained on how to implement effective operational procedures, believing instead that a piece of paper with a message on it would be effective. Their claim that a cleaning company should have been able to anticipate their failure and do a better job at operational assessment than RPI seems like a losing argument.

rcme 3y ago

Apparently they had a sign that explained how to mute the alarm. I think that’s going to work against them. They were aware that the alarm could trigger erroneously and didn’t fix the issue.

tekla 3y ago
Didn't read the article. The article clearly states the fridge was scheduled for repair, but due to COVID was delayed.
- rcme 3y ago
  Why does that matter? They knew it was broken but didn’t explicitly notify the cleaning company to avoid the fridge. Like what is the argument exactly? “I knew this was broken and didn’t tell you about it. Now you owe me money because the broken thing interfered with your work.”
  - justinclift 3y ago
    The sign was specifically for the cleaner, explaining the problem, plus instructing them not to clean the area. It also told them how to mute the alarm sound if it was annoying. In hind sight it would have been better to pre-contact the cleaning company and make sure the cleaner in question understood the situation. But that's only from knowing after the fact that the cleaner was hmmm... a friggin idiot. ;)
    - rcme 3y ago
      Do you really think the cleaner looks at every thing posted in the lab as he cleans? Very unlikely. Agreements require a meeting of the minds. The only idiots I see are the ones that lost decades of research.
      - justinclift 3y ago
        Depends on the sign. I guess I'm assuming it'd be "something obvious" but that could indeed be a really dumb assumption on my part. :)

dingosity 3y ago

It's research that's so valued they don't have a battery backup and don't page a grad student or two when the power goes off in the middle of the night. It's so important they don't explain the problem to people who have access to the lab saying "hey. look at this sign over here."

hinata08 3y ago
Just watch a few CSB videos (among others) on YT to understand how some companies are run, when it comes to functional safety and ergonomics. This is unfortunately not taught everywhere apparently. Long story short, some companies would have done exactly the same management for something that would cause Armageddon. In the 80s in Paris, a train crashed when some train driver thought he was helpful by responding to an alarm and fixing the brakes himself instead of calling maintenance, and actually turned off the brakes because a poorly design switch made it look like the circuit was open. Plot twist, he was found to be not guilty of anything because company procedure were not clear. So a mere freezer probably went under the radar of management. In this case, a power loss could have been accidental as well. What if a fuse went out, or some unrelated piece if equipment caused an upstream GFCI to trip ? How can you trust a fridge and its power supply to protect 1M dollars for 20+ years, if 20 years worth of samples were destroyed? Power was just bound to be lost at some point in time, over a period of a few decades. (It even happens in datacenters) And some UPS would cost up to 2000$ ? Up to 500 more for the warning system to send a text ? IMO, functional safety should be a staple in university.
- jstarfish 3y ago
  > How can you trust a fridge and its power supply to protect 1M dollars for 20+ years, if 20 years worth of samples were destroyed? Power was just bound to be lost at some point in time, over a period of a few decades. (It even happens in datacenters) This very point makes the entire thing so ridiculous I'm not convinced this isn't the setup to outright fraud. Like a failing storeowner who starts making customers with children traverse the artisanal glasscraft section to get to the toy section. "Oopsies! You owe me $1M." I can't say this is about money (it's a university), but suspect maybe this project was doomed anyway and now its coordinator is staging sabotage to save face. The only thing that gives me pause is the thought that it's a little weird the cleaner was messing with the circuit breakers at all. The explanation offered makes little sense.

tonymillion 3y ago

So they keep millions of dollars worth of samples in a single freezer in a single location? The loss of the samples was an inevitability and regardless of who pulled the plug - a cleaner, a power cut or a faulty compressor they are solely responsible for the loss. Assuming a cleaner would read a note attached to a freezer, given they no doubt have to see lab notes all over the place - and probably have instructions not to move or read the notes - is completely delusional. IMO the blame falls squarely at the feet of the researchers and RPI and at worst the cleaning company should agree to terminate the contract (thus losing the $1.4 million per year deal they had).

smcg 3y ago
I mean, looking at a site now that sells cryogenic freezers, only one model goes colder than -86C, and this was going down to less than -110C. this is expensive and specialized equipment. I understand not having a second one but they needed more safety and redundancy on this one.
- tonymillion 3y ago
  The article mentions Fahrenheit (-112f), which means it was -80C, so the first one you found would have been "good enough". A quick google finds a -80C freezer for $14000 [0] a fraction of the $1million they've "lost". [0] [Fisherbrand™ Isotemp™ Ultra Low Temperature Freezer](https://www.fishersci.com/shop/products/isotemp-ultra-low-temperature-freezer-25/IUE30086FA# https://www.fishersci.com/shop/products/isotemp-ultra-low-te...)
  - scrps 3y ago
    While I agree RPI screwed the pooch on this one; I am surprised for $14k the damn power switch isnt under a locked panel/switch cover given what it is used for.

JoeAltmaier 3y ago

Reminds me of the company that sequestered private keys. Some idiot VP went to the vault, took out the master list and email'd it to a friend who had forgot their key. Gutting the entire value of the company and ruining it forever. People do the damndest things. Critical stuff needs a little extra care put in place or some fool will inevitably do the thing-that-must-not-be-done. Sometimes I have no confidence that a mature civilization is even possible if it's composed of human beings.

ilyt 3y ago
> Sometimes I have no confidence that a mature civilization is even possible if it's composed of human beings. Of course it is, just takes some lawyers and PR to change the meaning of "mature"
vinyl7 3y ago
> Sometimes I have no confidence that a mature civilization is even possible if it's composed of human beings. It boggles my mind that humans have survived for so long considering how suicidal and self destructive we are as a species
- kjs3 3y ago
  For every one of us monkeys who says "hey, y'all...watch this!" and does something insanely stupid, there are always at least a couple who take a step back, make sure we're out of the blast radius, and think "this guy is going to make a great warning to the others"...thus preserving the species for another day.
tangjurine 3y ago
What company?

DoItToMe81 3y ago

If he was willing to risk all this, imagine just how fucking annoying that alarm must have been.

tangjurine 3y ago

The guy said he was trying to be helpful after hearing the annoying alarm by turning the electricity on. Either he's really dumb, or he just doesn't care about the consequences and then makes up some bs excuse after. My money is on the latter.