3 ms·
VFX sysadmin here > Setup alerting if you care about your data Yes. and make sure that you have the correct signal to noise ratio. Otherwise you won't act on
by KaiserPro 6y ago
VFX sysadmin here
> Setup alerting if you care about your data
Yes. and make sure that you have the correct signal to noise ratio. Otherwise you won't act on it.
> scrub everything
Yes, this too.
One thing that I would add is avoid mdadm like the plague. Its tricky, poorly documented and requires a whole bunch of other crap tools to make it work.
ZFS or proper hardware raid, there is no inbetween.
On the subject of loosing a raid during rebuild, I've had that, but I can't be sure that it was because of simultaneous disk failures. The raid in question was a 60 drive array with 8TB drives. There were 4 raid 7 groups with 4 spare drivers.
After each rebuild, a new drive would fail. over the space of two weeks we changed about 6 drives. Because the raid was under huge pressure, it didn't have the time needed to rebuild (24 hour rebuild time jumps up to 96 hours)
In the end we migrated the data to another array.
- derefr 6y ago> One thing that I would add is avoid mdadm like the plague. Its tricky, poorly documented and requires a whole bunch of other crap tools to make it work. Do you feel that Linux MDRAID is a bad technology, or just a bad UX? I.e., are MDRAID-based appliances (like many NASes)—where all the sharp edges have been smoothed away—still bad in some fundamental way?
- joana035 6y agomdadm is just fine. YMMV depending on the distro you are using and your knowledge/know-how about storage. I recommend to try it out with loop devices, play around and build your own opinion about it.
- KaiserPro 6y agoOh no, its a fine technology, its just useless documentation and to make it really work you need LVM, which is just horrific to deal with. I should say that I've worked proffeshionally with ZFS, mdamd/pv/lvm gluster(ugh) lustre(ok) clustered xfs and GPFS. Along with a bunch of appliances. mdadm/pv/lvm is about the bottom three. After gluster and ceph.
- core-questions 6y agoI find LVM2 to be easy to work with, but the weird marriage between MD and LVM confuses me, so I just use "hardware" raid beneath LVM. (For server OS installs that I can recreate trivially, and not for actual data, which lives on a proper SAN because I value my employment)
- joana035 6y agoHow come mdadm needs lvm? Do you mean device mapper?
- KaiserPro 6y agoI don't mind device mapper, for what I used it for, multipath scsi. mdadm doesn't need lvm. But if you want snapshots, or to have flexible partitions (anything like the flexibility of zfs) you need lvm. Also for the longest time redhat used to force lvm on you by default. I assume this was to help with "flexibility" In practice, on virtual machines, or anything back with decent storage, I'd remove the partition table and just run on the block device directly.
- louwrentius 6y ago> One thing that I would add is avoid mdadm like the plague. Its tricky, poorly documented and requires a whole bunch of other crap tools to make it work. Strange, I have exactly the opposite view. I think ZFS is exceptionally well documented and easy to use. But MDADM is just as simple. What other 'crap tools' are you talking about? Regarding your RAID60 experience: it could be a bad batch of drives. Also, I wonder if they had a good solid 24H break-in period with a significant load to weed out bad drives before the array was taken into production.
- KaiserPro 6y agoZFS's documentation is as good as the docs for expensive storage. (HEllo IBM redbooks, they are _wonderful_) To use mdadm properly you need lvm as well. Which means dealing with all the rubbish that come with it. (unrecoverable snapshots as a service? that might be fixed now.) > Regarding your RAID60 experience: it could be a bad batch of drives. Naaa it was the workload. We had 32 of these server that were largely identical, and bought in two batches. This one (it was actually two) server was running the blackhole fluid sim for interstella. These servers had all been up and running for well over a year. I should point out that with about 2000 drives for these servers, we would be replacing at least three disks a week. Normally they'd be spread evenly over all servers, it was unusual that this array ate ~7 drives in two weeks, with back to back rebuilds. Interestingly, the nearline machines which had 4x the drives, was much less likely to loose disks. (however they had a different usage pattern, virtually no random access, and it was one raid controller on the top 60 drive array, then three jbods daisy chained.)
- louwrentius 6y agoOk clear story, thanks. LVM is a different story. To be frank, my article was focused on the smb / home nas user. In enterprise environments, it’s a different story.
- bc4m 6y agoWhat is the mdadm use-case that requires lvm? I've used mdadm happily for over a decade and never used lvm.