3 ms·

Curious what sort of redundancy these systems have. I mean, you think of a web production environment... you don't just get an alarm, you typically have redund

Curious what sort of redundancy these systems have. I mean, you think of a web production environment... you don't just get an alarm, you typically have redundant systems you can turn on if primary servers fail. Seems like they only had one system running, and when it failed they were boned. Wonder what it would take to build more redundancy -- seems like samples that are this rare would deserve freezers inside of freezers? Multiple AC units for the room? Or is it the case that these samples can be easily restored by re-drilling in various places? Or is it the case that despite age and rarity, they don't have that much value after being catalogued?

jrs235 10y ago
Yeah. I would expect monitors and alarms on all freezers and a freezer inside a freezer.
zkms 10y ago
https://www.ualberta.ca/news-and-events/newsarticles/2017/april/freezer-failure-results-in-damage-to-ice-core-collection https://www.ualberta.ca/news-and-events/newsarticles/2017/ap... There's more details about the causes of the failure at that URL: > the refrigeration chillers shut down due to “high head pressure” conditions. Essentially, the chillers were not able to reject their heat through the condenser water system—heat instead of cold circulated through the freezer. > Compounding matters, the system monitoring the freezer temperatures failed due to a database corruption. The freezer’s computer system was actually sending out alarm signals that the temperature was rising, but those signals never made it to the university’s service provider or the on-campus control centre. To properly increase ability to respond to failure conditions use of an in-situ system (that doesn't depend on any additional infrastructure) that monitors the temperature before and after the evaporators and if they are adding heat rather than removing it (like what happened in this failure case), would be extremely useful -- indeed, this is one of the proposed changes: "The refrigeration system will also be modified to improve performance during failures, such as shutting down the evaporators in the event of a condenser failure." As for redundancy, a secondary set of chillers would provide less protection than expected -- they'd likely be running on the same condenser water system and exposed to the same conditions and abuse as each other. Since the melted cores were so expensive to get, implementing an independent refrigeration mechanism (the same way nuclear reactors have alternative cooling mechanisms) that isn't vulnerable to common-cause failures is an utterly sensible approach and should have been done. A method of providing such a backup source of refrigeration would be liquid nitrogen cooling -- universities/research labs tend to already have bulk cryogenic tanks of liquid nitrogen (that get refilled regularly from a Praxair or Air Liquide liquid nitrogen truck), so provision of LN2 is no issue. In the worst case, an additional LN2 tank would have to be installed to ensure that there's always enough LN2 on-site to last long enough until either alternative emergency refrigeration can be arranged (via emergency chiller rental) or the normal chillers can be repaired. Such a reserve tank would not incur prohibitive expenses and its normal gaseous boil-off can be used as gas nitrogen supply just like the main tank's boil-off. All that would be needed is a connection from the LN2 tank to the ice core storage facility, a proper set of evaporators/heat exchangers/fans that work with LN2 as cooling fluid, and also a control system that, upon detection of anomalous conditions, shuts off and isolates the normal chillers and activates the LN2 backup cooling system. There's very few common points of failure between the normal chillers and the LN2 system, and the LN2 system doesn't cool with electrical power and so wouldn't be affected by electrical outages.
- dbg31415 10y ago
  Great explanation, thanks!

jrs235 10y ago

Yeah. I would expect monitors and alarms on all freezers and a freezer inside a freezer.

zkms 10y ago

https://www.ualberta.ca/news-and-events/newsarticles/2017/april/freezer-failure-results-in-damage-to-ice-core-collection https://www.ualberta.ca/news-and-events/newsarticles/2017/ap... There's more details about the causes of the failure at that URL: > the refrigeration chillers shut down due to “high head pressure” conditions. Essentially, the chillers were not able to reject their heat through the condenser water system—heat instead of cold circulated through the freezer. > Compounding matters, the system monitoring the freezer temperatures failed due to a database corruption. The freezer’s computer system was actually sending out alarm signals that the temperature was rising, but those signals never made it to the university’s service provider or the on-campus control centre. To properly increase ability to respond to failure conditions use of an in-situ system (that doesn't depend on any additional infrastructure) that monitors the temperature before and after the evaporators and if they are adding heat rather than removing it (like what happened in this failure case), would be extremely useful -- indeed, this is one of the proposed changes: "The refrigeration system will also be modified to improve performance during failures, such as shutting down the evaporators in the event of a condenser failure." As for redundancy, a secondary set of chillers would provide less protection than expected -- they'd likely be running on the same condenser water system and exposed to the same conditions and abuse as each other. Since the melted cores were so expensive to get, implementing an independent refrigeration mechanism (the same way nuclear reactors have alternative cooling mechanisms) that isn't vulnerable to common-cause failures is an utterly sensible approach and should have been done. A method of providing such a backup source of refrigeration would be liquid nitrogen cooling -- universities/research labs tend to already have bulk cryogenic tanks of liquid nitrogen (that get refilled regularly from a Praxair or Air Liquide liquid nitrogen truck), so provision of LN2 is no issue. In the worst case, an additional LN2 tank would have to be installed to ensure that there's always enough LN2 on-site to last long enough until either alternative emergency refrigeration can be arranged (via emergency chiller rental) or the normal chillers can be repaired. Such a reserve tank would not incur prohibitive expenses and its normal gaseous boil-off can be used as gas nitrogen supply just like the main tank's boil-off. All that would be needed is a connection from the LN2 tank to the ice core storage facility, a proper set of evaporators/heat exchangers/fans that work with LN2 as cooling fluid, and also a control system that, upon detection of anomalous conditions, shuts off and isolates the normal chillers and activates the LN2 backup cooling system. There's very few common points of failure between the normal chillers and the LN2 system, and the LN2 system doesn't cool with electrical power and so wouldn't be affected by electrical outages.

dbg31415 10y ago
Great explanation, thanks!