5 ms·

> I suppose by now them and other hosting providers have taken that issue into account and are modifying their DC designs accordingly. Love your faith in human

> I suppose by now them and other hosting providers have taken that issue into account and are modifying their DC designs accordingly. Love your faith in humanity but I think we'd both be surprised about how little other hosting providers have changed after this incident.

KennyBlanken 5y ago
This setup is so far off standard practice it's shocking. Freeflowing does not excuse not having proper auto-closing dampers, triggered by the fire alarms or suppression system. Automatic power disconnects based on fire suppression system triggering are pretty standard. Most of the systems are fail-safe, meaning if the fire suppression panel loses power or is disconnected from the breakers feeding the room, the breakers trip off. That there were no automatic shutoffs or easily accessible breakers to disable the generators is shocking. Most NOCs I've visited, there are remote genset controls on one of the walls, often remote transfer switch controls as well. At the very least, on the way out the door someone in the NOC should have pushed the right buttons. I've never even heard of a commercial datacenter that didn't have fire suppression. Some companies go for preaction systems to reduce the chances of a "everything gets completely fucked by water" systems (gaseous systems are the gold standard, hugely expensive to install compared to water.) They had wood ceilings!? In my career in the US, I've been around numerous situations where it was pretty obvious the fire inspector was being paid off or at the very least had an extremely cordial relationship with the building management. My guess is that something similar was happening here. I don't understand how they managed to have insurance, or get the ISO certifications they claimed they had.
- HstryrsrBttn 5y ago
  Autoclosing is a must. What fire department didn't do the walkthrough? Here that is obligatory and when that happens you can be sure that even a carton sti left from the newest installation is reprimanded. The last room I took take of was accepted with protocol by an architect. The carpenter and electrition had to explain every single change they made and justify that fire protection was reasonable (e.g. if necessary an extra layer of plasterboard and plaster on every small bit that could ignite was mandatory to gain an est. 30min on top here and there) ISO: what apart from documentation is necessary for ISO? What Standard are you referring to?
hughrr 5y ago
I’ve been in a lot of major data centres in the UK and OVH is definitely a complete joke compared to the worst of them. Apart from Host Europe in Nottingham who no longer exist there AFAIK. That was worse than OVH.
- richardfey 5y ago
  How does it practically work? e.g. if I were OVH I just pick any building owner that gives me the cheapest deal, and such owner has absolutely zero concerns as well about safety? Or would OVH own or co-own the building? I am surprised that the landlord didn't have a virtuous conflict of interest in this case.
  - wongarsu 5y ago
    It's a massive multi-story datacenter used only by them, specifically designed and built for their passive cooling concept, surrounded by other datacenters used only by them. I would be very surprised if they didn't own the building
  - hughrr 5y ago
    They specified a price and it was built to that price point. All corners cut as requested.
- i_have_an_idea 5y ago
  > OVH is definitely a complete joke compared to the worst of them what leads you to say that? just curious
  - hughrr 5y ago
    Even the bad ones had proper fire suppression systems and electrical cut outs.
- bartvk 5y ago
  If you make such strong statements, I would expect that you back it up. For now, it reads as a shallow dismissal, and the guidelines talk about that: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
johannes1234321 5y ago
Authorities typically take fire safety serious. So if the fire department write that in their report, authorities typically will adapt their regulation and inspections.
- ta988 5y ago
  In US and Germany yes, in France? We shall see.

KennyBlanken 5y ago

This setup is so far off standard practice it's shocking. Freeflowing does not excuse not having proper auto-closing dampers, triggered by the fire alarms or suppression system. Automatic power disconnects based on fire suppression system triggering are pretty standard. Most of the systems are fail-safe, meaning if the fire suppression panel loses power or is disconnected from the breakers feeding the room, the breakers trip off. That there were no automatic shutoffs or easily accessible breakers to disable the generators is shocking. Most NOCs I've visited, there are remote genset controls on one of the walls, often remote transfer switch controls as well. At the very least, on the way out the door someone in the NOC should have pushed the right buttons. I've never even heard of a commercial datacenter that didn't have fire suppression. Some companies go for preaction systems to reduce the chances of a "everything gets completely fucked by water" systems (gaseous systems are the gold standard, hugely expensive to install compared to water.) They had wood ceilings!? In my career in the US, I've been around numerous situations where it was pretty obvious the fire inspector was being paid off or at the very least had an extremely cordial relationship with the building management. My guess is that something similar was happening here. I don't understand how they managed to have insurance, or get the ISO certifications they claimed they had.

HstryrsrBttn 5y ago
Autoclosing is a must. What fire department didn't do the walkthrough? Here that is obligatory and when that happens you can be sure that even a carton sti left from the newest installation is reprimanded. The last room I took take of was accepted with protocol by an architect. The carpenter and electrition had to explain every single change they made and justify that fire protection was reasonable (e.g. if necessary an extra layer of plasterboard and plaster on every small bit that could ignite was mandatory to gain an est. 30min on top here and there) ISO: what apart from documentation is necessary for ISO? What Standard are you referring to?

hughrr 5y ago

I’ve been in a lot of major data centres in the UK and OVH is definitely a complete joke compared to the worst of them. Apart from Host Europe in Nottingham who no longer exist there AFAIK. That was worse than OVH.

richardfey 5y ago
How does it practically work? e.g. if I were OVH I just pick any building owner that gives me the cheapest deal, and such owner has absolutely zero concerns as well about safety? Or would OVH own or co-own the building? I am surprised that the landlord didn't have a virtuous conflict of interest in this case.
- wongarsu 5y ago
  It's a massive multi-story datacenter used only by them, specifically designed and built for their passive cooling concept, surrounded by other datacenters used only by them. I would be very surprised if they didn't own the building
- hughrr 5y ago
  They specified a price and it was built to that price point. All corners cut as requested.
i_have_an_idea 5y ago
> OVH is definitely a complete joke compared to the worst of them what leads you to say that? just curious
- hughrr 5y ago
  Even the bad ones had proper fire suppression systems and electrical cut outs.
bartvk 5y ago
If you make such strong statements, I would expect that you back it up. For now, it reads as a shallow dismissal, and the guidelines talk about that: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

johannes1234321 5y ago

Authorities typically take fire safety serious. So if the fire department write that in their report, authorities typically will adapt their regulation and inspections.

ta988 5y ago
In US and Germany yes, in France? We shall see.