9 ms·

Cooling system leak led to Victorian Big Battery fire

110 points by bjowen 5y ago

RicoElectrico 5y ago
So, they'll have to take note from their nuclear colleagues' playbook? About passive safety.
- karlkloss 5y ago
  Why? The factory didn't burn down, and there was no widespread radioactive contamination. It's a battery pack, not a stash of U235 above the critical mass. Every recycling facility is more dangerous.
  - ashtonkem 5y ago
    Because fires are bad, and they can kill workers and shut down facilities?
- cotillion 5y ago
  Well, the conclusions seem to suggest they have passive safety. "The affected Megapacks failed safely despite total loss."
  - epistasis 5y ago
    I think that people understand the level of engineering and technology that goes into batteries and their designs these days. There’s a ton of talent, a ton of continuous innovation, it in the end it too often gets relegated to a black box for energy in and energy out. Which is in some ways a testament to the quality of the engineering.
bserge 5y ago
Should've used metal flanges instead of plastic heh
spockz 5y ago
> There were further contributory factors with the Megapack in question being switched into an off-line service mode, resulting in the protection systems being inactive. A short in an offline device? That seems weird. > A 24-hour delay in connecting the batteries to the supervisory control and data acquisition (SCADA) system also meant there was no active monitoring of the Megapack alarms. So a new pack was installed and connected but not monitored? What is the rationale behind that? Failures shouldn’t happen to new devices?
- jlg23 5y ago
  Department #1 does the hardware setup, signs off their forms and the next day department #2 integrates the new hardware into monitoring?
- michaelt 5y ago
  > So a new pack was installed and connected but not monitored? What is the rationale behind that? Any time a house is built, there's some time between the ceiling going up and the fire alarm installation being completed. Nothing wrong with that - if it's done properly and carefully.
tomalpha 5y ago
An extract from the summary page suggests a few things went wrong, but it seems like the fire was caused by liquid and electricity mixing: > ESV found a Megapack cooling system leak caused a short circuit resulting in overheating that led to a fire in a nearby battery compartment, which consequently damaged two Megapacks. > There were further contributory factors with the Megapack in question being switched into an off-line service mode, resulting in the protection systems being inactive. > A 24-hour delay in connecting the batteries to the supervisory control and data acquisition (SCADA) system also meant there was no active monitoring of the Megapack alarms.
- sitkack 5y ago
  I would assume that each megapack would have zigbee for telemetry which should be always transmitting, even if the pack were in the off state. And the cooling system should be continuously monitored for faults. Not just inspected after filling.
  - WJW 5y ago
    That it's hardware instead of software doesn't mean it can't be hacked together.
    - sitkack 5y ago
      Thermals are the number one aspect of battery engineering. For something that is to run under load in the Victorian heat, even more so. This isn't some weekend hackathon contest, Tesla is supposed to have the worlds best engineering. https://en.wikipedia.org/wiki/Tesla_Megapack https://en.wikipedia.org/wiki/Tesla_Megapack
      - WJW 5y ago
        Why would Tesla have the "worlds best" engineering? I don't think I have ever seen anyone ever claim that except for perhaps Tesla PR representatives. How would you even measure "best" in such a field? Cheapest? Best quality? Best value-for-money? Safest? Highest power density?
cube00 5y ago
> Shorter connection times to the SCADA system to help alert Tesla with specific alarms. Interesting that even after installation Tesla are still involved in operationally managing the megapacks.
- toomuchtodo 5y ago
  Tesla’s model is they both sell the Megapack system and then orchestrate it over its lifetime using their Autobidder platform. The owner is responsible for financing, integration with on site generation, and working with the local grid operator.
aetherspawn 5y ago
> "A new battery module isolation loss alarm has been added." This is the worst fault possible in an EV 101 scenario right here (touch metal -> instant death), so I'm surprised that with all Tesla's experience, they didn't consider an isolation leak as a serious fault. And I'm surprised that they allow the battery monitoring system to be disabled during servicing, especially the IMD (insulation monitoring device).
- londons_explore 5y ago
  I am constantly surprised how many systems have no isolation when it would be easy and cheap to add. For example, typical solar inverter systems don't have isolation between the AC and DC sides. That means any chafed wire can be deadly. They do at least have leakage detection, but set at a 300mA level, which probably won't save your life. Considering solar inverters already uses high frequency switching inside the per-string MPPT boost converter, it would have cost mere cents to put two windings on that inductor rather than one and get isolation with no downside. The cynic in me says manufacturers don't add safety features not required by law, even if zero cost, because in the future laws might be updated to require that safety feature, preventing reuse of hardware already sold, which is good for business.
  - baybal2 5y ago
    Aren't you are mistaking galvanic isolation with insulation?
    - aetherspawn 5y ago
      Galvanic isolation is useful because it means that you have to touch 2 specific points on the affected unit simultaneously to die, rather than 1 point. Galvanically isolated systems can detect leaks on both the positive and negative sides by using a dual channel IMD. Any insulation issue is bad, but this method of detection can usually detect it before it's able to instantly kill someone per-se. Electric vehicles are an example of a galvanically isolated system (from earth) with dual channel IMD, hence you can have massive leaks such as 8-20mA before it becomes a safety issue.
nraynaud 5y ago
how much would you ask to be paid to pressure test the coolant system of a giant bomb in situ?
william2021 5y ago
Short circuit that led to a fire in an electronic component
dhsysusbsjsi 5y ago
Reading this it’s a classic systems failure: multiple defensive barriers gone wrong through poor safety planning. It’s reminiscent of the Victorian ESSO gas plant accident https://en.wikipedia.org/wiki/Esso_Longford_gas_explosion https://en.wikipedia.org/wiki/Esso_Longford_gas_explosion
- krisoft 5y ago
  In what way is it reminding you of that incident? The paralels I can see: - both involve energy installations. - both happened in the same state. The differences I can see: - one involves a gas utility the other an electric one - one has killed and hurt workers while the other did not killed nor injured anyone. - one happened after several years of operation, the other happened during installation. - one left the state without energy service for 20 days, the other did not affect service at all. Don’t get me wrong. I love reading about how complex things go wrong, one of my favourite pastimes. So I thank you for your link. It just doesn’t seem that similar to me. Not sure you would be citing the same case for example have they not happened in the same state. If I have to choose a failure which the Big Battery fire reminds me of I would rather choose the Florida International University pedestrian bridge collapse. Why? It seems these batteries had adequate safety functions and monitoring planned in for their operation. Just these systems were not activated during the installation in the right order. Similarly with the bridge it seems they did calculations with the full bridge in place, but nobody seems to have checked if the intermediate steps during the construction will stay up too. Another similarility (to my untrained eyes) is the aim to install a system fast. The bridge project wanted to be a flagship for accelerated bridge building, and the big selling point of these Megapack installations is how fast they can be comissioned. With the bridge project we know that this demand for speediness was a contributing factor. Was it maybe also with the Big Battery fire too?
  - throwaway0a5e 5y ago
    >In what way is it reminding you of that incident? It's just in group signaling. "Hey this reminds me of <vaguely related thing>" is a more polite way of saying "look I know about X, I'm like you, gimme dat virtue points" You see this on literally every virtue points (i.e. up-vote) based form of social media that's big enough that any given participant can blend in with the crowd. The behavior is endemic these days you can't even post a picture of a rotten and falling down deck on the internet these days without some jerk derailing everything by dropping a link to the wikipedia page on the hyatt regency and then several more riding their coattails with quotes copied from the page. He probably doesn't even consciously realize he's doing it.
baybal2 5y ago
One advantage of LFP batteries is that they can survive much higher temperatures without degradation. If you can make battery packs air cooled, that's a big, big weight saver.
walrus01 5y ago
for people who might say 'oh, but these batteries are so dangerous!' How many cooling system leaks (radiator+water pump related plumbing) result in vehicle engine fires per year? I imagine the largest insurance companies have reasonably good data on this as a cause of total-loss of a vehicle. How many times have you seen in person, or seen a photo of a burned out RV that somebody pushed too hard up a mountain pass without keeping an eye on the engine temperature?
- DoingIsLearning 5y ago
  Read the doc, this has nothing to do with lithium batteries as a component. This was a System Requirements/Hazard Analysis failure. The fault lies square with Tesla. No amount of RV analogies can dilute this.
  - walrus01 5y ago
    The point was that plenty of other energy dense things with radiator-based liquid cooling loops lack proper monitoring systems. Not excusing Tesla but the actual number of heat producing things that catch on fire every year (in general) because of cooling pump/cooling loop failure is quite a lot. In many categories of equipment.
- driverdan 5y ago
  > How many cooling system leaks (radiator+water pump related plumbing) result in vehicle engine fires per year? Almost none. Overheating is highly unlikely to cause a fire. Fuel leaks cause fires.
- phaedrus 5y ago
  You have a gross misconception of the relative temperatures of components in internal combustion engines, and you're using a faulty conclusion based on the same misunderstanding as supposed supporting evidence. I assure you an RV or other gas/diesel vehicle does NOT suddenly go "pop!" and produce an engine fire when the needle reaches the right hand side of the temperature gauge. The temperature gauge warns you when the coolant is at risk of boiling; it's not a warning that the engine is about to spontaneously combust.
userbinator 5y ago
From the title I first thought this would be about a battery fire that happened in the late 1800s.
- bjowen 5y ago
  In a much cooler reality, it is, and it’s the other Tesla.

110 points by bjowen 5y ago

RicoElectrico 5y ago

So, they'll have to take note from their nuclear colleagues' playbook? About passive safety.

karlkloss 5y ago
Why? The factory didn't burn down, and there was no widespread radioactive contamination. It's a battery pack, not a stash of U235 above the critical mass. Every recycling facility is more dangerous.
- ashtonkem 5y ago
  Because fires are bad, and they can kill workers and shut down facilities?
cotillion 5y ago
Well, the conclusions seem to suggest they have passive safety. "The affected Megapacks failed safely despite total loss."
- epistasis 5y ago
  I think that people understand the level of engineering and technology that goes into batteries and their designs these days. There’s a ton of talent, a ton of continuous innovation, it in the end it too often gets relegated to a black box for energy in and energy out. Which is in some ways a testament to the quality of the engineering.

bserge 5y ago

Should've used metal flanges instead of plastic heh

spockz 5y ago

> There were further contributory factors with the Megapack in question being switched into an off-line service mode, resulting in the protection systems being inactive. A short in an offline device? That seems weird. > A 24-hour delay in connecting the batteries to the supervisory control and data acquisition (SCADA) system also meant there was no active monitoring of the Megapack alarms. So a new pack was installed and connected but not monitored? What is the rationale behind that? Failures shouldn’t happen to new devices?

jlg23 5y ago
Department #1 does the hardware setup, signs off their forms and the next day department #2 integrates the new hardware into monitoring?
michaelt 5y ago
> So a new pack was installed and connected but not monitored? What is the rationale behind that? Any time a house is built, there's some time between the ceiling going up and the fire alarm installation being completed. Nothing wrong with that - if it's done properly and carefully.

tomalpha 5y ago

An extract from the summary page suggests a few things went wrong, but it seems like the fire was caused by liquid and electricity mixing: > ESV found a Megapack cooling system leak caused a short circuit resulting in overheating that led to a fire in a nearby battery compartment, which consequently damaged two Megapacks. > There were further contributory factors with the Megapack in question being switched into an off-line service mode, resulting in the protection systems being inactive. > A 24-hour delay in connecting the batteries to the supervisory control and data acquisition (SCADA) system also meant there was no active monitoring of the Megapack alarms.

sitkack 5y ago
I would assume that each megapack would have zigbee for telemetry which should be always transmitting, even if the pack were in the off state. And the cooling system should be continuously monitored for faults. Not just inspected after filling.
- WJW 5y ago
  That it's hardware instead of software doesn't mean it can't be hacked together.
  - sitkack 5y ago
    Thermals are the number one aspect of battery engineering. For something that is to run under load in the Victorian heat, even more so. This isn't some weekend hackathon contest, Tesla is supposed to have the worlds best engineering. https://en.wikipedia.org/wiki/Tesla_Megapack https://en.wikipedia.org/wiki/Tesla_Megapack
    - WJW 5y ago
      Why would Tesla have the "worlds best" engineering? I don't think I have ever seen anyone ever claim that except for perhaps Tesla PR representatives. How would you even measure "best" in such a field? Cheapest? Best quality? Best value-for-money? Safest? Highest power density?

cube00 5y ago

> Shorter connection times to the SCADA system to help alert Tesla with specific alarms. Interesting that even after installation Tesla are still involved in operationally managing the megapacks.

toomuchtodo 5y ago
Tesla’s model is they both sell the Megapack system and then orchestrate it over its lifetime using their Autobidder platform. The owner is responsible for financing, integration with on site generation, and working with the local grid operator.

aetherspawn 5y ago

> "A new battery module isolation loss alarm has been added." This is the worst fault possible in an EV 101 scenario right here (touch metal -> instant death), so I'm surprised that with all Tesla's experience, they didn't consider an isolation leak as a serious fault. And I'm surprised that they allow the battery monitoring system to be disabled during servicing, especially the IMD (insulation monitoring device).

londons_explore 5y ago
I am constantly surprised how many systems have no isolation when it would be easy and cheap to add. For example, typical solar inverter systems don't have isolation between the AC and DC sides. That means any chafed wire can be deadly. They do at least have leakage detection, but set at a 300mA level, which probably won't save your life. Considering solar inverters already uses high frequency switching inside the per-string MPPT boost converter, it would have cost mere cents to put two windings on that inductor rather than one and get isolation with no downside. The cynic in me says manufacturers don't add safety features not required by law, even if zero cost, because in the future laws might be updated to require that safety feature, preventing reuse of hardware already sold, which is good for business.
- baybal2 5y ago
  Aren't you are mistaking galvanic isolation with insulation?
  - aetherspawn 5y ago
    Galvanic isolation is useful because it means that you have to touch 2 specific points on the affected unit simultaneously to die, rather than 1 point. Galvanically isolated systems can detect leaks on both the positive and negative sides by using a dual channel IMD. Any insulation issue is bad, but this method of detection can usually detect it before it's able to instantly kill someone per-se. Electric vehicles are an example of a galvanically isolated system (from earth) with dual channel IMD, hence you can have massive leaks such as 8-20mA before it becomes a safety issue.

nraynaud 5y ago

how much would you ask to be paid to pressure test the coolant system of a giant bomb in situ?

william2021 5y ago

Short circuit that led to a fire in an electronic component

dhsysusbsjsi 5y ago

Reading this it’s a classic systems failure: multiple defensive barriers gone wrong through poor safety planning. It’s reminiscent of the Victorian ESSO gas plant accident https://en.wikipedia.org/wiki/Esso_Longford_gas_explosion https://en.wikipedia.org/wiki/Esso_Longford_gas_explosion

krisoft 5y ago
In what way is it reminding you of that incident? The paralels I can see: - both involve energy installations. - both happened in the same state. The differences I can see: - one involves a gas utility the other an electric one - one has killed and hurt workers while the other did not killed nor injured anyone. - one happened after several years of operation, the other happened during installation. - one left the state without energy service for 20 days, the other did not affect service at all. Don’t get me wrong. I love reading about how complex things go wrong, one of my favourite pastimes. So I thank you for your link. It just doesn’t seem that similar to me. Not sure you would be citing the same case for example have they not happened in the same state. If I have to choose a failure which the Big Battery fire reminds me of I would rather choose the Florida International University pedestrian bridge collapse. Why? It seems these batteries had adequate safety functions and monitoring planned in for their operation. Just these systems were not activated during the installation in the right order. Similarly with the bridge it seems they did calculations with the full bridge in place, but nobody seems to have checked if the intermediate steps during the construction will stay up too. Another similarility (to my untrained eyes) is the aim to install a system fast. The bridge project wanted to be a flagship for accelerated bridge building, and the big selling point of these Megapack installations is how fast they can be comissioned. With the bridge project we know that this demand for speediness was a contributing factor. Was it maybe also with the Big Battery fire too?
- throwaway0a5e 5y ago
  >In what way is it reminding you of that incident? It's just in group signaling. "Hey this reminds me of <vaguely related thing>" is a more polite way of saying "look I know about X, I'm like you, gimme dat virtue points" You see this on literally every virtue points (i.e. up-vote) based form of social media that's big enough that any given participant can blend in with the crowd. The behavior is endemic these days you can't even post a picture of a rotten and falling down deck on the internet these days without some jerk derailing everything by dropping a link to the wikipedia page on the hyatt regency and then several more riding their coattails with quotes copied from the page. He probably doesn't even consciously realize he's doing it.

baybal2 5y ago

One advantage of LFP batteries is that they can survive much higher temperatures without degradation. If you can make battery packs air cooled, that's a big, big weight saver.

walrus01 5y ago

for people who might say 'oh, but these batteries are so dangerous!' How many cooling system leaks (radiator+water pump related plumbing) result in vehicle engine fires per year? I imagine the largest insurance companies have reasonably good data on this as a cause of total-loss of a vehicle. How many times have you seen in person, or seen a photo of a burned out RV that somebody pushed too hard up a mountain pass without keeping an eye on the engine temperature?

DoingIsLearning 5y ago
Read the doc, this has nothing to do with lithium batteries as a component. This was a System Requirements/Hazard Analysis failure. The fault lies square with Tesla. No amount of RV analogies can dilute this.
- walrus01 5y ago
  The point was that plenty of other energy dense things with radiator-based liquid cooling loops lack proper monitoring systems. Not excusing Tesla but the actual number of heat producing things that catch on fire every year (in general) because of cooling pump/cooling loop failure is quite a lot. In many categories of equipment.
driverdan 5y ago
> How many cooling system leaks (radiator+water pump related plumbing) result in vehicle engine fires per year? Almost none. Overheating is highly unlikely to cause a fire. Fuel leaks cause fires.
phaedrus 5y ago
You have a gross misconception of the relative temperatures of components in internal combustion engines, and you're using a faulty conclusion based on the same misunderstanding as supposed supporting evidence. I assure you an RV or other gas/diesel vehicle does NOT suddenly go "pop!" and produce an engine fire when the needle reaches the right hand side of the temperature gauge. The temperature gauge warns you when the coolant is at risk of boiling; it's not a warning that the engine is about to spontaneously combust.

userbinator 5y ago

From the title I first thought this would be about a battery fire that happened in the late 1800s.

bjowen 5y ago
In a much cooler reality, it is, and it’s the other Tesla.