7 ms·

Investigation report on the OVH data centre fire in Strasbourg on 2021-03-10

60 points by speedgoose 4y ago

speedgoose 4y ago
> Could a water leak on an electronic board of an inverter be the cause of the beginning of the disaster? The report is in French, but you can look at the pictures or use an online translator. https://deepl.com https://deepl.com usually provides much more convincing French-English translations compared to Google Translate.
- naniwaduni 4y ago
  Note that "more convincing" isn't necessarily a good thing; the deception of fluency can totally be used to convince you of something wrong.
  - lucb1e 4y ago
    I've had good experiences with deepl, but certainly a good point. Translating definitely has an interpretation aspect to it. I'd be curious if anyone tried tricky texts on different translators and how they fared. I'm not enough of a linguist myself to know of different categories of tricks (to give a fair assessment rather than things randomly coming to mind), or what even the jargon for such things would be to look them up.
    - naniwaduni 4y ago
      Well, this went around my circles the other day: https://twitter.com/Xythar/status/1405660710382706694 https://twitter.com/Xythar/status/1405660710382706694 It turns out that the "tricks" don't necessarily have to get very sophisticated, because, well, the target user of machine translation services can't understand the source text. (I mean, the fr-en language pair is probably better off than ja-en, but I don't know enough French to know whether it can be trusted!)
      - Mordisquitos 4y ago
        I'm not sure what to make of this.
        deleted 4y ago
        [deleted]
      - lucb1e 4y ago
        > in truth it's like 10 times more likely to just make complete shit up if it doesn't understand the source I use this to learn German. Instead of translating a text EN->DE and not practicing any writing skills, I'll write my best attempt at German and see if it's understandable by running it through the DE->EN or DE->NL translator. (In cases where I care about the quality, I'll then patch up the English/Dutch if necessary, run it NL/EN->DE, and use that version.) For this purpose, I'm glad that it makes a best guess at what my broken German must mean, and it usually does a fair job (easy to say because I know what I meant, so no validation issues there). Of course this is not great for every use-case, for example it would be better if it additionally displayed confidence (e.g. slightly graying out subsentences below 80% confidence), but it also has advantages to make a good guess at the meaning of the source. And when I put in something unintelligible, unintelligible stuff comes out. It's not really just making something up, at least not in my experience with EN/DE/NL. No idea what happened there with that Japanese example.
      - mananaysiempre 4y ago
        As an aside, there seems to have been a dramatic change in Google Translate’s underlying approach several years ago: before it, the fr>en and zh>en translations were reasonable (if stilted and occasionally ungrammatical) while both de>en and ja>en absolutely sucked (just made no sense whatsoever over groups of more than three or so words); I suspect the difficulty being the “global” transformations needed to translate SOV to SVO word order. It would be very interesting to know what they did (there are old-style statistical approaches that involve learning pairs of corresponding syntax trees, but I didn’t get the impression they were practical? might have been an artifact of the original DoD research being focused on zh and ru, which are both SVO—though ar is VSO and fa is SOV...).
      - 4y ago
nraynaud 4y ago
I can't imagine the phone call to Germany: "Hi, it's the clown neighbor again, we are sending you smoke, probably laced with lead. Sweet Dreams."
- Kuinox 4y ago
  Well that's fair, they send us their coal fumes all the year.
  - sofixa 4y ago
    It'd be funny if it weren't so tragic. Germany really should have shut down coal (i know it's easier said than done and there's a significant industry around it, all the more reason to start with it and not postpone) before nuclear.
    - walrus01 4y ago
      home of the famous BAGGER 288 https://en.wikipedia.org/wiki/Bagger_288 https://en.wikipedia.org/wiki/Bagger_288
adamredwoods 4y ago
Interesting read (Google translated). >> Safety lessons in building design >> In the field of building design, we will retain two safety lessons. >> First of all, the requirements applicable to battery charging rooms, when they are located inside a building, require a sufficient degree of fire resistance to prevent its propagation to the rest of the building. The existing regulations already seem complete to us, and the OHV accident does not call their technical relevance into question. >> However, two configurations, in the current state of the regulations, deserve particular attention: >> - When the batteries used are not likely to generate hydrogen during charging (if lead batteries are now mainly used in energy storage in data centers, lithium technology offers one more alternative more competitive which tends to develop); >> - Or when these load rooms are located outside. >> On the first point, the BEA-RI considers that the prescription relating to the constructive provisions should also concern the other battery technologies for which electrical failure and thermal runaway cannot be physically ruled out. This type of failure can lead to major fires and justify specific construction measures. >> On the second point (outdoor charging rooms), the BEA-RI recalls the recommendations issued in its report MTE-BEARI-2021-004 on the battery container fire in Perles and Castelet (09) >> Finally, the report points out that protecting the battery room is not sufficient, given the outbreak of fire at the level of the inverter:
- spydum 4y ago
  used to build and work in a couple DCs in the 2000 era.. can confirm the battery room was legit the most terrifying room we had (multistory buildings, so they were inside). something about the stored potential energy and risk of physical hazard always tripped my spidey senses. ATS panels also..
  - jacquesm 4y ago
    Also: power for large phone switch boards. Those battery banks are very impressive.
  - andybak 4y ago
    > ATS panels also.. https://www.shentongroup.co.uk/blog/what-is-an-ats-panel-automatic-transfer-switch/ https://www.shentongroup.co.uk/blog/what-is-an-ats-panel-aut...
molyss 4y ago
translation of the main points (of the linked article, not the full 43-pages BEA-RI report) : Fire started simultaneously in a battery and and in a power inverter. It doesn't look like there's any comment on why/how it started on either and how/why it started on both at the same time. Batteries were lead-acid, not Lithium-Ion (lead-acid is supposed to be less of a fire-hazard). There's timestamped video of the start of the fire. Alarm was triggered within less than a minute. Someone was at the initial fire location within 2 minutes. Building was evacuated within 4 minutes. Fire Department called within 7 minutes of fire start. On site 17 minutes later. Power company is called within 17 minutes of the 1st fire alarm. Power company couldn't cut power from the substation because of fire risk (and delay to get approval from OVH, which owned the substation). The upstream substation is shut down 80minutes after the fire started. The fire-engulfed building still has power for another 98 minutes. Fire is considered under control 3h after that (6h after it started), extinguished another 3h later. There was significant concern that the building would collapse. There was no fire sprinklers or fire-extinguishing mechanism in the building itself. Water supply was insufficient (single fire hydrant delivering 16k gallons/h). Now water reserve, or pumping mechanism (the Rhein was super close). Only after 2h were the firefighter able to throw water at the blaze. Before that, there was still electrical power on site. Recommendations: Battery storage already have some regulation WRT fire hazard. Charging equipment don't. Apply the same rules for both improve the ability to cut power from a site There's a bunch of question that pop in my mind : Concerning the construction, how common is it to have a 1-h fire resistance ? In the report, they say the floors are made out of wood with a treatment to allow for 1h firewall, and the "internal structure" has a 1h "fire stability". That sounds awfully low to me, especially considering that the fire took 6h to control, 10h to clear. What's "R-5" concrete structure ? The power is provided through 2 redundant 20kV AC lines. Then, the power can follow one of 3 path : directly fed into the hardware if it's "clean" enough corrected to be cleaner before being fed into the hardware. converted to DC before being fed into the batteries. If the batteries are needed, the power is first converted back into AC before hitting the servers, which will convert it back to DC. My question : Why don't we convert all the power to DC in a central location before feeding DC power to the servers ? I would expect some significant saving costs in DC->AC conversion, no need for AC "cleaning", and the ability to extract the AC->DC conversion out of the servers (increasing density, and removing some more heat close to the servers). I'm sure it's not a new "idea", I'm just curious as to why it's not a good one. On the report itself, it looks like the presence of battery makes electrical cutoff much harder. If you cut the main power lines, the batteries and power generators take over, so your DC is still powered.
walrus01 4y ago
I'm not sure why all the complaints and concern, seems like a fairly successful server migration to the cloud to me.
- danw1979 4y ago
  Took me a moment, but I got it in the end.
MrStonedOne 4y ago

60 points by speedgoose 4y ago

> Could a water leak on an electronic board of an inverter be the cause of the beginning of the disaster? The report is in French, but you can look at the pictures or use an online translator. https://deepl.com https://deepl.com usually provides much more convincing French-English translations compared to Google Translate.

naniwaduni 4y ago
Note that "more convincing" isn't necessarily a good thing; the deception of fluency can totally be used to convince you of something wrong.
- lucb1e 4y ago
  I've had good experiences with deepl, but certainly a good point. Translating definitely has an interpretation aspect to it. I'd be curious if anyone tried tricky texts on different translators and how they fared. I'm not enough of a linguist myself to know of different categories of tricks (to give a fair assessment rather than things randomly coming to mind), or what even the jargon for such things would be to look them up.
  - naniwaduni 4y ago
    Well, this went around my circles the other day: https://twitter.com/Xythar/status/1405660710382706694 https://twitter.com/Xythar/status/1405660710382706694 It turns out that the "tricks" don't necessarily have to get very sophisticated, because, well, the target user of machine translation services can't understand the source text. (I mean, the fr-en language pair is probably better off than ja-en, but I don't know enough French to know whether it can be trusted!)
    - Mordisquitos 4y ago
      I'm not sure what to make of this.
      - deleted 4y ago
        [deleted]
    - lucb1e 4y ago
      > in truth it's like 10 times more likely to just make complete shit up if it doesn't understand the source I use this to learn German. Instead of translating a text EN->DE and not practicing any writing skills, I'll write my best attempt at German and see if it's understandable by running it through the DE->EN or DE->NL translator. (In cases where I care about the quality, I'll then patch up the English/Dutch if necessary, run it NL/EN->DE, and use that version.) For this purpose, I'm glad that it makes a best guess at what my broken German must mean, and it usually does a fair job (easy to say because I know what I meant, so no validation issues there). Of course this is not great for every use-case, for example it would be better if it additionally displayed confidence (e.g. slightly graying out subsentences below 80% confidence), but it also has advantages to make a good guess at the meaning of the source. And when I put in something unintelligible, unintelligible stuff comes out. It's not really just making something up, at least not in my experience with EN/DE/NL. No idea what happened there with that Japanese example.
    - mananaysiempre 4y ago
      As an aside, there seems to have been a dramatic change in Google Translate’s underlying approach several years ago: before it, the fr>en and zh>en translations were reasonable (if stilted and occasionally ungrammatical) while both de>en and ja>en absolutely sucked (just made no sense whatsoever over groups of more than three or so words); I suspect the difficulty being the “global” transformations needed to translate SOV to SVO word order. It would be very interesting to know what they did (there are old-style statistical approaches that involve learning pairs of corresponding syntax trees, but I didn’t get the impression they were practical? might have been an artifact of the original DoD research being focused on zh and ru, which are both SVO—though ar is VSO and fa is SOV...).
    - 4y ago

nraynaud 4y ago

I can't imagine the phone call to Germany: "Hi, it's the clown neighbor again, we are sending you smoke, probably laced with lead. Sweet Dreams."

Kuinox 4y ago
Well that's fair, they send us their coal fumes all the year.
- sofixa 4y ago
  It'd be funny if it weren't so tragic. Germany really should have shut down coal (i know it's easier said than done and there's a significant industry around it, all the more reason to start with it and not postpone) before nuclear.
  - walrus01 4y ago
    home of the famous BAGGER 288 https://en.wikipedia.org/wiki/Bagger_288 https://en.wikipedia.org/wiki/Bagger_288

adamredwoods 4y ago

Interesting read (Google translated). >> Safety lessons in building design >> In the field of building design, we will retain two safety lessons. >> First of all, the requirements applicable to battery charging rooms, when they are located inside a building, require a sufficient degree of fire resistance to prevent its propagation to the rest of the building. The existing regulations already seem complete to us, and the OHV accident does not call their technical relevance into question. >> However, two configurations, in the current state of the regulations, deserve particular attention: >> - When the batteries used are not likely to generate hydrogen during charging (if lead batteries are now mainly used in energy storage in data centers, lithium technology offers one more alternative more competitive which tends to develop); >> - Or when these load rooms are located outside. >> On the first point, the BEA-RI considers that the prescription relating to the constructive provisions should also concern the other battery technologies for which electrical failure and thermal runaway cannot be physically ruled out. This type of failure can lead to major fires and justify specific construction measures. >> On the second point (outdoor charging rooms), the BEA-RI recalls the recommendations issued in its report MTE-BEARI-2021-004 on the battery container fire in Perles and Castelet (09) >> Finally, the report points out that protecting the battery room is not sufficient, given the outbreak of fire at the level of the inverter:

spydum 4y ago
used to build and work in a couple DCs in the 2000 era.. can confirm the battery room was legit the most terrifying room we had (multistory buildings, so they were inside). something about the stored potential energy and risk of physical hazard always tripped my spidey senses. ATS panels also..
- jacquesm 4y ago
  Also: power for large phone switch boards. Those battery banks are very impressive.
- andybak 4y ago
  > ATS panels also.. https://www.shentongroup.co.uk/blog/what-is-an-ats-panel-automatic-transfer-switch/ https://www.shentongroup.co.uk/blog/what-is-an-ats-panel-aut...

molyss 4y ago

translation of the main points (of the linked article, not the full 43-pages BEA-RI report) : Fire started simultaneously in a battery and and in a power inverter. It doesn't look like there's any comment on why/how it started on either and how/why it started on both at the same time. Batteries were lead-acid, not Lithium-Ion (lead-acid is supposed to be less of a fire-hazard). There's timestamped video of the start of the fire. Alarm was triggered within less than a minute. Someone was at the initial fire location within 2 minutes. Building was evacuated within 4 minutes. Fire Department called within 7 minutes of fire start. On site 17 minutes later. Power company is called within 17 minutes of the 1st fire alarm. Power company couldn't cut power from the substation because of fire risk (and delay to get approval from OVH, which owned the substation). The upstream substation is shut down 80minutes after the fire started. The fire-engulfed building still has power for another 98 minutes. Fire is considered under control 3h after that (6h after it started), extinguished another 3h later. There was significant concern that the building would collapse. There was no fire sprinklers or fire-extinguishing mechanism in the building itself. Water supply was insufficient (single fire hydrant delivering 16k gallons/h). Now water reserve, or pumping mechanism (the Rhein was super close). Only after 2h were the firefighter able to throw water at the blaze. Before that, there was still electrical power on site. Recommendations: Battery storage already have some regulation WRT fire hazard. Charging equipment don't. Apply the same rules for both improve the ability to cut power from a site There's a bunch of question that pop in my mind : Concerning the construction, how common is it to have a 1-h fire resistance ? In the report, they say the floors are made out of wood with a treatment to allow for 1h firewall, and the "internal structure" has a 1h "fire stability". That sounds awfully low to me, especially considering that the fire took 6h to control, 10h to clear. What's "R-5" concrete structure ? The power is provided through 2 redundant 20kV AC lines. Then, the power can follow one of 3 path : directly fed into the hardware if it's "clean" enough corrected to be cleaner before being fed into the hardware. converted to DC before being fed into the batteries. If the batteries are needed, the power is first converted back into AC before hitting the servers, which will convert it back to DC. My question : Why don't we convert all the power to DC in a central location before feeding DC power to the servers ? I would expect some significant saving costs in DC->AC conversion, no need for AC "cleaning", and the ability to extract the AC->DC conversion out of the servers (increasing density, and removing some more heat close to the servers). I'm sure it's not a new "idea", I'm just curious as to why it's not a good one. On the report itself, it looks like the presence of battery makes electrical cutoff much harder. If you cut the main power lines, the batteries and power generators take over, so your DC is still powered.

walrus01 4y ago

I'm not sure why all the complaints and concern, seems like a fairly successful server migration to the cloud to me.

danw1979 4y ago
Took me a moment, but I got it in the end.

MrStonedOne 4y ago