3 ms·

> One thing that I would add is avoid mdadm like the plague. Its tricky, poorly documented and requires a whole bunch of other crap tools to make it work. Stra

> One thing that I would add is avoid mdadm like the plague. Its tricky, poorly documented and requires a whole bunch of other crap tools to make it work. Strange, I have exactly the opposite view. I think ZFS is exceptionally well documented and easy to use. But MDADM is just as simple. What other 'crap tools' are you talking about? Regarding your RAID60 experience: it could be a bad batch of drives. Also, I wonder if they had a good solid 24H break-in period with a significant load to weed out bad drives before the array was taken into production.

KaiserPro 6y ago
ZFS's documentation is as good as the docs for expensive storage. (HEllo IBM redbooks, they are _wonderful_) To use mdadm properly you need lvm as well. Which means dealing with all the rubbish that come with it. (unrecoverable snapshots as a service? that might be fixed now.) > Regarding your RAID60 experience: it could be a bad batch of drives. Naaa it was the workload. We had 32 of these server that were largely identical, and bought in two batches. This one (it was actually two) server was running the blackhole fluid sim for interstella. These servers had all been up and running for well over a year. I should point out that with about 2000 drives for these servers, we would be replacing at least three disks a week. Normally they'd be spread evenly over all servers, it was unusual that this array ate ~7 drives in two weeks, with back to back rebuilds. Interestingly, the nearline machines which had 4x the drives, was much less likely to loose disks. (however they had a different usage pattern, virtually no random access, and it was one raid controller on the top 60 drive array, then three jbods daisy chained.)
- louwrentius 6y ago
  Ok clear story, thanks. LVM is a different story. To be frank, my article was focused on the smb / home nas user. In enterprise environments, it’s a different story.
- bc4m 6y ago
  What is the mdadm use-case that requires lvm? I've used mdadm happily for over a decade and never used lvm.
  - nine_k 6y ago
    If you want snapshots on XFS, you want lvm, else (AFAICT) you can only make a snapshot while the XFS volume is offline.

KaiserPro 6y ago

ZFS's documentation is as good as the docs for expensive storage. (HEllo IBM redbooks, they are _wonderful_) To use mdadm properly you need lvm as well. Which means dealing with all the rubbish that come with it. (unrecoverable snapshots as a service? that might be fixed now.) > Regarding your RAID60 experience: it could be a bad batch of drives. Naaa it was the workload. We had 32 of these server that were largely identical, and bought in two batches. This one (it was actually two) server was running the blackhole fluid sim for interstella. These servers had all been up and running for well over a year. I should point out that with about 2000 drives for these servers, we would be replacing at least three disks a week. Normally they'd be spread evenly over all servers, it was unusual that this array ate ~7 drives in two weeks, with back to back rebuilds. Interestingly, the nearline machines which had 4x the drives, was much less likely to loose disks. (however they had a different usage pattern, virtually no random access, and it was one raid controller on the top 60 drive array, then three jbods daisy chained.)

louwrentius 6y ago
Ok clear story, thanks. LVM is a different story. To be frank, my article was focused on the smb / home nas user. In enterprise environments, it’s a different story.
bc4m 6y ago
What is the mdadm use-case that requires lvm? I've used mdadm happily for over a decade and never used lvm.
- nine_k 6y ago
  If you want snapshots on XFS, you want lvm, else (AFAICT) you can only make a snapshot while the XFS volume is offline.