6 ms·

It's a great thing for startups that OVH provides servers at such an amazing low cost. Yes, there is always a risk that someone will make a mistake in the build

It's a great thing for startups that OVH provides servers at such an amazing low cost. Yes, there is always a risk that someone will make a mistake in the building design. There's a chance that eliminating some redundancies increases the possibility of a failure. There's always a chance that something bad will go wrong. However, this isn't just a matter of Hanlon's razor (incompetence vs malice), but more of a matter of an intelligent guesstimate of risk versus a lack of knowledge in some areas (wasn't this OVH's very first datacenter?), and a strong focus on reducing costs. Perhaps the latter went too far, and definitely some obvious mistakes were made by not having a universal power cut off of some sort, but dealing with the amount of power on tap in a datacenter is always dangerous, even when there is no fire at all. I'm not saying we need to give OVH a complete pass on this. I'm just saying that there are a lot of extenuating circumstances and, except for the power cut off, it's not clear that OVH made any choices due to extreme negligence or cost-cutting. In other words, they didn't do anything immoral. At worst, it appears that (even from the most anti-OVH party here), this was just a mistake in the design of a new (at the time) style of datacenter, and it did work properly for many years before there was a problem. Making a mistake is not immoral.

ddaalluu2 5y ago
"Amazing low cost"? They're one of the most expensive data center operators I know. Hetzner is about half the price. But neither can have more than 3 HDD per machine which is just absurd.
- dspillett 5y ago
  > neither can have more than 3 HDD per machine which is just absurd Not sure where you get that information from. Some of Hetzner's “auction” machines have 10 drives, many of OVH's offerings support more than three (even some of the budget range, Kimsufi, are 4x2T).
  - ddaalluu2 5y ago
    I'm not talking about auctions. I'm talking about regular servers. And I get those facts from ACTUALLY FUCKING TRYING TO CONFIGURE MORR THAN 3 HDDS on those machines. You go ahead and try it before talking out your ass and downvoting.
- MikusR 5y ago
  From https://www.hetzner.com/dedicated-rootserver?drives_count_from=16 https://www.hetzner.com/dedicated-rootserver?drives_count_fr... 2 x 3.84 TB NVMe SSD and 14 x 16 TB SATA HDD
  - ddaalluu2 5y ago
    Not talking about storage servers either
- Kuinox 5y ago
  I pay my OVH potato server 8€/month. 2TB SSD, ridiculous small CPU, 100MB up/down without any cost.
  - pmontra 5y ago
    2TB SSD? Please could you point me to an OVH web page with that offer? I looked for it and I didn't find anything.
    - Kuinox 5y ago
      Woops I meant HDD, can't edit it anymore. Won't make a lot of difference, the CPU is so bad it would be your bottleneck here. It's a baremetal sever available on their lowcost brand: https://www.kimsufi.com/en/ https://www.kimsufi.com/en/
- dhx 5y ago
  Hetzner Falkenstein was toured recently by a YouTuber and I didn't spot any fire suppression systems in the video [1]. OVH SBG1 (which was partially destroyed by the fire that wiped out SBG2) used shipping containers which also didn't appear to have any fire suppression systems [2]. By contrast, the typical data centers people know which have fire suppression systems appear to have much of their key electrical equipment (and control systems) located in the same area or adjoining rooms of the same facility [3] [4] [5] [6]. In fact, there is even a video recording of a UPS failure [7] showing a lucky case where no one was injured and not too much damage was caused. The employee lingered in the room when they should have immediately left at the first sign of danger. Arc flashes are a scary possibility as shown in [8] and [9] because of the need to switch multiple megawatts of electricity through complex power systems that includes UPS battery banks and automatic diesel generators. There are video recordings demonstrating how fire suppression systems work [10] [11] and a description of how a data center would be designed to respond to a fire (including closing ventilation dampers) [12] [13]. I'm sure fire suppression systems are not cheap, but in the grand scheme of a data center full of millions of dollars of equipment (not to mention cost of customer downtime) surely it would make sense to install them. [1] https://www.youtube.com/watch?v=5eo8nz_niiM https://www.youtube.com/watch?v=5eo8nz_niiM [2] https://www.youtube.com/watch?v=Gua9j4BZKKg https://www.youtube.com/watch?v=Gua9j4BZKKg [3] https://youtu.be/qUmLnSEVVDw?t=716 https://youtu.be/qUmLnSEVVDw?t=716 [4] https://youtu.be/LYncuYp0UVo?t=184 https://youtu.be/LYncuYp0UVo?t=184 [5] https://www.youtube.com/watch?v=LeFtwtvy4Wc https://www.youtube.com/watch?v=LeFtwtvy4Wc [6] https://www.youtube.com/watch?v=pGthey0Q1dw https://www.youtube.com/watch?v=pGthey0Q1dw [7] https://www.youtube.com/watch?v=Lhlhj-_7Rrc https://www.youtube.com/watch?v=Lhlhj-_7Rrc [8] https://www.youtube.com/watch?v=ipW4D0yQyME https://www.youtube.com/watch?v=ipW4D0yQyME [9] https://youtu.be/6hpE5LYj-CY?t=34 https://youtu.be/6hpE5LYj-CY?t=34 [10] https://www.youtube.com/watch?v=hjii88Jv6AU https://www.youtube.com/watch?v=hjii88Jv6AU [11] https://www.youtube.com/watch?v=SjZKsKXL-hI https://www.youtube.com/watch?v=SjZKsKXL-hI [12] https://www.youtube.com/watch?v=b4Jxs3h-3ZU https://www.youtube.com/watch?v=b4Jxs3h-3ZU [13] https://www.youtube.com/watch?v=zb-5fU-ILgo https://www.youtube.com/watch?v=zb-5fU-ILgo
  - dx034 5y ago
    > Hetzner Falkenstein was toured recently by a YouTuber and I didn't spot any fire suppression systems in the video [1]. OVH SBG1 (which was partially destroyed by the fire that wiped out SBG2) used shipping containers which also didn't appear to have any fire suppression systems [2]. They can definitely shut the vents at the top so that the problem with air circulation during a fire wouldn't appear. Furthermore, their dcs are only on one level, so it couldn't spread upwards. I don't know if they got an external fuse/switch, but it seems likely that there's something between substation and the individual dcs. As for fire suppresion, I tried to spot one but couldn't either. But he never shows all parts of the DC, it could be in a part that's not shown. However, with the design they use, if they can ensure that a fire is contained within one DC, the damage would probably be comparably low, especially if you consider the space they have between the racks.
- alexdumitru 5y ago
  I have multiple servers at Hetzner with 10 drives each.
  - ddaalluu2 5y ago
    Not talking about storage servers. New regular ax ex px series. No more than 3 HDDs not SSDs.
mihaaly 5y ago
Everything works well, until not. When it is worked for years without problem, but also without correcting initial sources of risks (learning the business after the clueless first years) that's like learning while driving that the trunk is full of flammable fluid but driving on as "nothing bad happened before". Buildings should be used within the safety margins of those and prepared for certain type of extrimeties, especially fire. We do not put risky operations into a construction that could not handle or mitigate potential risks (no electricity cutoff, not enough fire extunguising material, no cut off of intense ventillation). Operating a bakery in a barn without alterations comes to mind.
wk_end 5y ago
Where fire’s concerned, I do think all mistakes (failure to take reasonable precautions, and it sounds like this is the case) really are either negligent or immoral. The costs you save for yourself and your customers don’t factor in the externalities that will impact third parties in the event of a fire - namely, risk of damage to surrounding property, and risk of injury or death for the people who have to put that fire out.
- wongarsu 5y ago
  In this case at least the neighboring buildings are both from OVH as well and the only thing realistically at risk from spreading fire is a rail yard. Of course that doesn't excuse the design at all. Putting employees and firefighters at risk isn't ok, and I'm kind of baffled that they were allowed to operate like that. https://www.google.de/maps/@48.5848973,7.7971161,3a,75y,26.04h,83.57t/data=!3m6!1e1!3m4!1sybi8btFl3vvP8CokZsoE_A!2e0!7i16384!8i8192 https://www.google.de/maps/@48.5848973,7.7971161,3a,75y,26.0...
- throwaway0a5e 5y ago
  - throwaway2048 5y ago
    not every single conversation about doing better is a slippery slope until we are all slaves of the fire marshal.
  - dang 5y ago
    Please don't cross into personal attack. We ban accounts that do that and you can make your substantive points without swipes. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - throwaway0a5e 5y ago
      I don't think any self aware extremist is going to feel attacked by the phrase "extremist minority". They know where their beliefs stand relative to the population at large.
StreamBright 5y ago
This is why it is silly to compare OVH to AWS. Apples to oranges.
mabbo 5y ago
Did OVH make it clear to customers that the saved money was coming from lack of safety? I've never worked with them so I don't know. But it seems to me that if you're advertising yourself as equivalent to a competitor only they have fire suppression and you don't, you are obligated to bring that up. "We save you money because if anything goes wrong, your servers and all the critical data on them will be melted."

ddaalluu2 5y ago

"Amazing low cost"? They're one of the most expensive data center operators I know. Hetzner is about half the price. But neither can have more than 3 HDD per machine which is just absurd.

dspillett 5y ago
> neither can have more than 3 HDD per machine which is just absurd Not sure where you get that information from. Some of Hetzner's “auction” machines have 10 drives, many of OVH's offerings support more than three (even some of the budget range, Kimsufi, are 4x2T).
- ddaalluu2 5y ago
  I'm not talking about auctions. I'm talking about regular servers. And I get those facts from ACTUALLY FUCKING TRYING TO CONFIGURE MORR THAN 3 HDDS on those machines. You go ahead and try it before talking out your ass and downvoting.
MikusR 5y ago
From https://www.hetzner.com/dedicated-rootserver?drives_count_from=16 https://www.hetzner.com/dedicated-rootserver?drives_count_fr... 2 x 3.84 TB NVMe SSD and 14 x 16 TB SATA HDD
- ddaalluu2 5y ago
  Not talking about storage servers either
Kuinox 5y ago
I pay my OVH potato server 8€/month. 2TB SSD, ridiculous small CPU, 100MB up/down without any cost.
- pmontra 5y ago
  2TB SSD? Please could you point me to an OVH web page with that offer? I looked for it and I didn't find anything.
  - Kuinox 5y ago
    Woops I meant HDD, can't edit it anymore. Won't make a lot of difference, the CPU is so bad it would be your bottleneck here. It's a baremetal sever available on their lowcost brand: https://www.kimsufi.com/en/ https://www.kimsufi.com/en/
dhx 5y ago
Hetzner Falkenstein was toured recently by a YouTuber and I didn't spot any fire suppression systems in the video [1]. OVH SBG1 (which was partially destroyed by the fire that wiped out SBG2) used shipping containers which also didn't appear to have any fire suppression systems [2]. By contrast, the typical data centers people know which have fire suppression systems appear to have much of their key electrical equipment (and control systems) located in the same area or adjoining rooms of the same facility [3] [4] [5] [6]. In fact, there is even a video recording of a UPS failure [7] showing a lucky case where no one was injured and not too much damage was caused. The employee lingered in the room when they should have immediately left at the first sign of danger. Arc flashes are a scary possibility as shown in [8] and [9] because of the need to switch multiple megawatts of electricity through complex power systems that includes UPS battery banks and automatic diesel generators. There are video recordings demonstrating how fire suppression systems work [10] [11] and a description of how a data center would be designed to respond to a fire (including closing ventilation dampers) [12] [13]. I'm sure fire suppression systems are not cheap, but in the grand scheme of a data center full of millions of dollars of equipment (not to mention cost of customer downtime) surely it would make sense to install them. [1] https://www.youtube.com/watch?v=5eo8nz_niiM https://www.youtube.com/watch?v=5eo8nz_niiM [2] https://www.youtube.com/watch?v=Gua9j4BZKKg https://www.youtube.com/watch?v=Gua9j4BZKKg [3] https://youtu.be/qUmLnSEVVDw?t=716 https://youtu.be/qUmLnSEVVDw?t=716 [4] https://youtu.be/LYncuYp0UVo?t=184 https://youtu.be/LYncuYp0UVo?t=184 [5] https://www.youtube.com/watch?v=LeFtwtvy4Wc https://www.youtube.com/watch?v=LeFtwtvy4Wc [6] https://www.youtube.com/watch?v=pGthey0Q1dw https://www.youtube.com/watch?v=pGthey0Q1dw [7] https://www.youtube.com/watch?v=Lhlhj-_7Rrc https://www.youtube.com/watch?v=Lhlhj-_7Rrc [8] https://www.youtube.com/watch?v=ipW4D0yQyME https://www.youtube.com/watch?v=ipW4D0yQyME [9] https://youtu.be/6hpE5LYj-CY?t=34 https://youtu.be/6hpE5LYj-CY?t=34 [10] https://www.youtube.com/watch?v=hjii88Jv6AU https://www.youtube.com/watch?v=hjii88Jv6AU [11] https://www.youtube.com/watch?v=SjZKsKXL-hI https://www.youtube.com/watch?v=SjZKsKXL-hI [12] https://www.youtube.com/watch?v=b4Jxs3h-3ZU https://www.youtube.com/watch?v=b4Jxs3h-3ZU [13] https://www.youtube.com/watch?v=zb-5fU-ILgo https://www.youtube.com/watch?v=zb-5fU-ILgo
- dx034 5y ago
  > Hetzner Falkenstein was toured recently by a YouTuber and I didn't spot any fire suppression systems in the video [1]. OVH SBG1 (which was partially destroyed by the fire that wiped out SBG2) used shipping containers which also didn't appear to have any fire suppression systems [2]. They can definitely shut the vents at the top so that the problem with air circulation during a fire wouldn't appear. Furthermore, their dcs are only on one level, so it couldn't spread upwards. I don't know if they got an external fuse/switch, but it seems likely that there's something between substation and the individual dcs. As for fire suppresion, I tried to spot one but couldn't either. But he never shows all parts of the DC, it could be in a part that's not shown. However, with the design they use, if they can ensure that a fire is contained within one DC, the damage would probably be comparably low, especially if you consider the space they have between the racks.
alexdumitru 5y ago
I have multiple servers at Hetzner with 10 drives each.
- ddaalluu2 5y ago
  Not talking about storage servers. New regular ax ex px series. No more than 3 HDDs not SSDs.

mihaaly 5y ago

Everything works well, until not. When it is worked for years without problem, but also without correcting initial sources of risks (learning the business after the clueless first years) that's like learning while driving that the trunk is full of flammable fluid but driving on as "nothing bad happened before". Buildings should be used within the safety margins of those and prepared for certain type of extrimeties, especially fire. We do not put risky operations into a construction that could not handle or mitigate potential risks (no electricity cutoff, not enough fire extunguising material, no cut off of intense ventillation). Operating a bakery in a barn without alterations comes to mind.

wk_end 5y ago

Where fire’s concerned, I do think all mistakes (failure to take reasonable precautions, and it sounds like this is the case) really are either negligent or immoral. The costs you save for yourself and your customers don’t factor in the externalities that will impact third parties in the event of a fire - namely, risk of damage to surrounding property, and risk of injury or death for the people who have to put that fire out.

wongarsu 5y ago
In this case at least the neighboring buildings are both from OVH as well and the only thing realistically at risk from spreading fire is a rail yard. Of course that doesn't excuse the design at all. Putting employees and firefighters at risk isn't ok, and I'm kind of baffled that they were allowed to operate like that. https://www.google.de/maps/@48.5848973,7.7971161,3a,75y,26.04h,83.57t/data=!3m6!1e1!3m4!1sybi8btFl3vvP8CokZsoE_A!2e0!7i16384!8i8192 https://www.google.de/maps/@48.5848973,7.7971161,3a,75y,26.0...
throwaway0a5e 5y ago
- throwaway2048 5y ago
  not every single conversation about doing better is a slippery slope until we are all slaves of the fire marshal.
- dang 5y ago
  Please don't cross into personal attack. We ban accounts that do that and you can make your substantive points without swipes. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - throwaway0a5e 5y ago
    I don't think any self aware extremist is going to feel attacked by the phrase "extremist minority". They know where their beliefs stand relative to the population at large.

StreamBright 5y ago

This is why it is silly to compare OVH to AWS. Apples to oranges.

mabbo 5y ago

Did OVH make it clear to customers that the saved money was coming from lack of safety? I've never worked with them so I don't know. But it seems to me that if you're advertising yourself as equivalent to a competitor only they have fire suppression and you don't, you are obligated to bring that up. "We save you money because if anything goes wrong, your servers and all the critical data on them will be melted."