4 ms·

I've relied on hardware RAID over one hardware failure, for the first and last time. We were running some services on a stack of older Dell servers with mostly

I've relied on hardware RAID over one hardware failure, for the first and last time. We were running some services on a stack of older Dell servers with mostly identical hardware. The RAID controller and mainboard on one of them let the magic smoke out. We (a sysadmin team of three) promptly decided to move the RAID array to another unused server in the stack in order to avoid a lengthy reinstall and manual restoring of daily backups – this happened before the almost-one-click provisioning of devops and virtualization tools was the norm. The other server lacked the RAID license dongle that was on a pin header on the mainboard, so we just tried to cannibalize it from the server corpse that barely had stopped smoking. The new mainboard promptly rejected the license because the service tag differed, and the only way to change it was to install Windows 95 or 98 and run the tag changer tool. In addition, we had to downgrade the controller firmware to match that of the old one. One nightlong unattended rebuild later, the server was up and running the next morning. When we upgraded the server hardware a bit later, we migrated it over to mdraid and haven't had any problems with hardware failures since. Moral of the story: If you are depending on certain hardware features, you need to have a cold spare available with identical hardware and firmware. If you cannot afford one, it's better to either use a software implementation, have a really good service contract with the vendor or just let a VM provider take care of your infrastructure.

oarsinsync 6y ago
> Moral of the story: If you are depending on certain hardware features, you need to have a cold spare available with identical hardware and firmware. If you cannot afford one, it's better to either use a software implementation, have a really good service contract with the vendor or just let a VM provider take care of your infrastructure. +1 for this. Hardware raid absolutely has its place, but if you expect to be able to rebuild an array without restoring from a backup, you absolutely need to have spare parts that are tested in advance. Also, RAID is not a backup. RAID provides high(er) availability. You need to have backups still!
- anilakar 6y ago
  >Also, RAID is not a backup. RAID provides high(er) availability. You need to have backups still! Absolutely true. In this case we underestimated the workload caused by the service tag and firmware version issues and the fact that the new controller wanted to check and rebuild the array, when the disks themselves should have been fine. In hindsight it might have been faster to reinstall the OS and services plus restore the backups.
  - ryandrake 6y ago
    I always thought the rise of RansomWare would put a nail in the ol' "RAID as poor-man's backup" coffin, but guess not!
- ggeorgovassilis 6y ago
  > Also, RAID is not a backup. You are absolutely right, but there are use-cases where backup is too expensive, so investing into robustness and availability is the only option left.
  - barrkel 6y ago
    Backups give you a third thing: immutability. You can have a nice redundant storage for local performance and resilience, even geographic replication for resilience against a single site loss, but you need snapshots of some kind to protect against replicated destructive updates.
  - nix23 6y ago
    >backup is too expensive, so investing into robustness and availability is the only option left. No please don't say something like that if you love your data. Raid is for the availability of your data, a Backup is the secure storage/versioning of you data, two completely different things.
    - ggeorgovassilis 6y ago
      My love for my data has a price tag :-) If I afford backup with this amount, I need to cut down on other expenses in my life.
      - nix23 6y ago
        Have a look at some used LTO4 tape-drives ~40$ they can store up to 1.6TB per Tape (compressed) and are the perfect thing to offline your data.
        ggeorgovassilis 6y ago
        Thanks. I looked at the specifications; it's not clear how that would connect to the server. All SATA ports are taken and USB is too slow (estimated 72 hours) for a full backup and would require 6-8 tape changes.
miahi 6y ago
This is the reason I now use a 3rd party RAID controller on my HP Microserver Gen 8 - I have 3 RAID cards with the same hardware and firmware, and tested that the RAID config can be imported from one to the other.
m463 6y ago
I ran into the same (type of) problem years ago. I think I had a raid card with two mirrored drives. Thing is when one drive failed, I couldn't just pull the data off because the remaining drive could only be accessed by that raid controller.
rkagerer 6y ago
An untested disaster recovery or backup plan is a bad plan. Sorry you had to learn that the hard way. Agreed that identical or confirmed working spare components are important. I've migrated RAID arrays through several generations of Areca cards successfully and even tested back and forth between models. I like them. Also remember RAID is not backup; keep a second copy of your data around (offsite if you're serious).
pabs3 6y ago
IIRC the Linux kernel RAID infrastructure supports various on-disk data structures generated by various hardware RAID controllers, so probably it would be easier to recover from your situation in 2020.
- agapon 6y ago
  I am not sure but I think that it is for various "soft RAIDs" like those that come with AHCI / SATA controllers (including integrated ones). True hardware RAID controlelr probably require more magic.

oarsinsync 6y ago

> Moral of the story: If you are depending on certain hardware features, you need to have a cold spare available with identical hardware and firmware. If you cannot afford one, it's better to either use a software implementation, have a really good service contract with the vendor or just let a VM provider take care of your infrastructure. +1 for this. Hardware raid absolutely has its place, but if you expect to be able to rebuild an array without restoring from a backup, you absolutely need to have spare parts that are tested in advance. Also, RAID is not a backup. RAID provides high(er) availability. You need to have backups still!

anilakar 6y ago
>Also, RAID is not a backup. RAID provides high(er) availability. You need to have backups still! Absolutely true. In this case we underestimated the workload caused by the service tag and firmware version issues and the fact that the new controller wanted to check and rebuild the array, when the disks themselves should have been fine. In hindsight it might have been faster to reinstall the OS and services plus restore the backups.
- ryandrake 6y ago
  I always thought the rise of RansomWare would put a nail in the ol' "RAID as poor-man's backup" coffin, but guess not!
ggeorgovassilis 6y ago
> Also, RAID is not a backup. You are absolutely right, but there are use-cases where backup is too expensive, so investing into robustness and availability is the only option left.
- barrkel 6y ago
  Backups give you a third thing: immutability. You can have a nice redundant storage for local performance and resilience, even geographic replication for resilience against a single site loss, but you need snapshots of some kind to protect against replicated destructive updates.
- nix23 6y ago
  >backup is too expensive, so investing into robustness and availability is the only option left. No please don't say something like that if you love your data. Raid is for the availability of your data, a Backup is the secure storage/versioning of you data, two completely different things.
  - ggeorgovassilis 6y ago
    My love for my data has a price tag :-) If I afford backup with this amount, I need to cut down on other expenses in my life.
    - nix23 6y ago
      Have a look at some used LTO4 tape-drives ~40$ they can store up to 1.6TB per Tape (compressed) and are the perfect thing to offline your data.
      - ggeorgovassilis 6y ago
        Thanks. I looked at the specifications; it's not clear how that would connect to the server. All SATA ports are taken and USB is too slow (estimated 72 hours) for a full backup and would require 6-8 tape changes.

miahi 6y ago

This is the reason I now use a 3rd party RAID controller on my HP Microserver Gen 8 - I have 3 RAID cards with the same hardware and firmware, and tested that the RAID config can be imported from one to the other.

m463 6y ago

I ran into the same (type of) problem years ago. I think I had a raid card with two mirrored drives. Thing is when one drive failed, I couldn't just pull the data off because the remaining drive could only be accessed by that raid controller.

rkagerer 6y ago

An untested disaster recovery or backup plan is a bad plan. Sorry you had to learn that the hard way. Agreed that identical or confirmed working spare components are important. I've migrated RAID arrays through several generations of Areca cards successfully and even tested back and forth between models. I like them. Also remember RAID is not backup; keep a second copy of your data around (offsite if you're serious).

pabs3 6y ago

IIRC the Linux kernel RAID infrastructure supports various on-disk data structures generated by various hardware RAID controllers, so probably it would be easier to recover from your situation in 2020.

agapon 6y ago
I am not sure but I think that it is for various "soft RAIDs" like those that come with AHCI / SATA controllers (including integrated ones). True hardware RAID controlelr probably require more magic.