4 ms·

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this.

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down. Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. These entire datacenter failures should be anticipated, but to expect them is a bit too easy I think. There are plenty of hosters who don't have this stuff even once for the last decade in their only datacenter. I do not find it strange to expect or even demand that level but to protect yourself if it happens in any case if that fits your specific project and budget. Edit; OK meant that raid-5 remark in the same context as the hosting; it can and does happen but it shouldn't; you should plan for a contingency but expect it goes far. We never had it (1000s of hard-drive, decades of hosting, millions of sites) and so we plan for it with backups; if it happens it will take some downtime but it costs next to nothing over time to do that. If we expected it, we would need to take far different measures. And we had less downtime in a decade than aws AZ had in the past months. I have a problem with the word 'expect'.

dylan604 5y ago
>And that won't happen HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner. All drives were purchased at the same time, and happened to all come from a single batch from the manufacture. The manufacture confirmed this via serial numbers, and admitted they had an issue during production. All drives were replaced and at a larger volume size. TL;DR: Drives will fail, and manufacturing issues happend. Don't buy all of your drives in an array from the same batch! It will happen. To say it won't is just pure inexeperience.
- tluyben2 5y ago
  Guess I was lucky, we ran a lot of these over the decades when things were far more unreliable than now and never experienced anything like that. Manufacturing issues, sure, but we always had everything we bought run on stress for 48 hours and see if that killed it, if it didn't, it didn't usually break anymore (I have many of the machines from mid to end 2000s still and they don't have diskfailures now while they ran for many years).
  - dannyw 5y ago
    The older hdds imo are more reliable than the newer ones. They're pushing for density, and mitigating the inheritly higher sensitivity through aggressive error correction.
- Tempest1981 5y ago
  Would like to know the manufacturer and model.
  - dylan604 5y ago
    Sorry, this was back in 2006-2007 time frame. I have no idea on model numbers as that's just not information I ever cared to commit to memory.
    - tluyben2 5y ago
      My time frame is about since end 90s to now. I saw more failures in general in the olden days. I have quite a lot of rented servers at traditional hosters or Colo that have not been down outside kernel security updates for 10+ years. No hardware issues. I am now swapping them out for new servers which cost less for more, but hardware wise they have been pounder hard for 10 years without issues. All hotswap raid5 drives, so when broken, they or us just fixed it without downtime.
phone8675309 5y ago
> Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. There are plenty of situations where this might happen if they’re in your room: a lightning strike can cause a surge that causes the disks to fry, a thief might break in and steal your system, your house might burn down, an earthquake could cause your disks to crash, a flood could destroy the machines, and a sinkhole could open up and swallow your house. You may laugh at some of these as being improbable, but I have seen _all_ of these take out systems between my times in Florida (lightning, thief, sinkhole, and flood) and California (earthquake and house fire). The fix for this is the same fix as being proposed by the parent post - putting physical space between the two systems so if one place become unavailable you still have a backup.
- greiskul 5y ago
  I have had a job where my small, internal tool, for debugging purposes, had to be deployed to a minimum of 3 datacenters. I had 2 of them in the US and one in Europe, and was asked to move one of the US ones to a datacenter that was in another coast, cause who knows, maybe an earthquake will knock off all of the US west coast. That is the paranoia level necessary to achieve crazy high uptime.
acdha 5y ago
> If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. Here are some examples where that happened: 1. Drive manufacturer had a hardware issue affecting a certain production batch, causing failures pretty reliably after a certain number of power-on hours. A friend learned the hard way that his request to have mixed drives in his RAID array wasn’t followed. 2. AC issues showed a problem with airflow, causing one row to get enough warmer that faults were happening faster than the RAID rebuild time. 3. UPS took out a couple racks by cycling power off and on repeatedly until the hardware failed. No, these aren’t common but they were very hard to recover from because even if some of the drives were usable you couldn’t trust them. One interesting dynamic of the public clouds are that you tend to have better bounds on the maximum outage duration, which is an interesting trade off compared to several incidents I’ve seen where the downtime stretched into weeks due to replacement delays or manual rebuild processes.
AtNightWeCode 5y ago
"Won't happen". The 40,000 hours of runtime bug did happen. I would recommend people to take backups and store them offline or at least isolated from the main storage.
- tluyben2 5y ago
  Sure, I plan for it but I do not expect it. And it never happened for me over decades. But I did plan for it, just not in the way the parent said and it did cost me far less.
  - tluyben2 5y ago
    Good to know ;) still think I am lucky; have had very little harddisks fail across 1000s. And only very few unrecoverable (had to restore from backup) failures; those weren't HD failures but software failures; hds were fine.
8note 5y ago
More generally, any correlation between two items gives potential for a correlated failure. Same manufacturer, same disk space, same location, same operator, same maintenance schedule, same legal jurisdiction, same planet, you name it, and there's a common failure to match

dylan604 5y ago

>And that won't happen HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner. All drives were purchased at the same time, and happened to all come from a single batch from the manufacture. The manufacture confirmed this via serial numbers, and admitted they had an issue during production. All drives were replaced and at a larger volume size. TL;DR: Drives will fail, and manufacturing issues happend. Don't buy all of your drives in an array from the same batch! It will happen. To say it won't is just pure inexeperience.

tluyben2 5y ago
Guess I was lucky, we ran a lot of these over the decades when things were far more unreliable than now and never experienced anything like that. Manufacturing issues, sure, but we always had everything we bought run on stress for 48 hours and see if that killed it, if it didn't, it didn't usually break anymore (I have many of the machines from mid to end 2000s still and they don't have diskfailures now while they ran for many years).
- dannyw 5y ago
  The older hdds imo are more reliable than the newer ones. They're pushing for density, and mitigating the inheritly higher sensitivity through aggressive error correction.
Tempest1981 5y ago
Would like to know the manufacturer and model.
- dylan604 5y ago
  Sorry, this was back in 2006-2007 time frame. I have no idea on model numbers as that's just not information I ever cared to commit to memory.
  - tluyben2 5y ago
    My time frame is about since end 90s to now. I saw more failures in general in the olden days. I have quite a lot of rented servers at traditional hosters or Colo that have not been down outside kernel security updates for 10+ years. No hardware issues. I am now swapping them out for new servers which cost less for more, but hardware wise they have been pounder hard for 10 years without issues. All hotswap raid5 drives, so when broken, they or us just fixed it without downtime.

phone8675309 5y ago

> Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. There are plenty of situations where this might happen if they’re in your room: a lightning strike can cause a surge that causes the disks to fry, a thief might break in and steal your system, your house might burn down, an earthquake could cause your disks to crash, a flood could destroy the machines, and a sinkhole could open up and swallow your house. You may laugh at some of these as being improbable, but I have seen _all_ of these take out systems between my times in Florida (lightning, thief, sinkhole, and flood) and California (earthquake and house fire). The fix for this is the same fix as being proposed by the parent post - putting physical space between the two systems so if one place become unavailable you still have a backup.

greiskul 5y ago
I have had a job where my small, internal tool, for debugging purposes, had to be deployed to a minimum of 3 datacenters. I had 2 of them in the US and one in Europe, and was asked to move one of the US ones to a datacenter that was in another coast, cause who knows, maybe an earthquake will knock off all of the US west coast. That is the paranoia level necessary to achieve crazy high uptime.

acdha 5y ago

> If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. Here are some examples where that happened: 1. Drive manufacturer had a hardware issue affecting a certain production batch, causing failures pretty reliably after a certain number of power-on hours. A friend learned the hard way that his request to have mixed drives in his RAID array wasn’t followed. 2. AC issues showed a problem with airflow, causing one row to get enough warmer that faults were happening faster than the RAID rebuild time. 3. UPS took out a couple racks by cycling power off and on repeatedly until the hardware failed. No, these aren’t common but they were very hard to recover from because even if some of the drives were usable you couldn’t trust them. One interesting dynamic of the public clouds are that you tend to have better bounds on the maximum outage duration, which is an interesting trade off compared to several incidents I’ve seen where the downtime stretched into weeks due to replacement delays or manual rebuild processes.

AtNightWeCode 5y ago

"Won't happen". The 40,000 hours of runtime bug did happen. I would recommend people to take backups and store them offline or at least isolated from the main storage.

tluyben2 5y ago
Sure, I plan for it but I do not expect it. And it never happened for me over decades. But I did plan for it, just not in the way the parent said and it did cost me far less.
- tluyben2 5y ago
  Good to know ;) still think I am lucky; have had very little harddisks fail across 1000s. And only very few unrecoverable (had to restore from backup) failures; those weren't HD failures but software failures; hds were fine.

8note 5y ago

More generally, any correlation between two items gives potential for a correlated failure. Same manufacturer, same disk space, same location, same operator, same maintenance schedule, same legal jurisdiction, same planet, you name it, and there's a common failure to match