3 ms·

Can anyone chime in with their experience using dm-integrity? I looked into it a while back for home RAID and recall seeing some limitations that made me steer

Can anyone chime in with their experience using dm-integrity? I looked into it a while back for home RAID and recall seeing some limitations that made me steer clear, possibly related to LUKS, or array expansion (I'm using mdadm RAID6 and I periodically grow my arrays). But that was a while ago, and I don't remember what the issues were. This time I promise to keep notes.

martijnvds 5y ago
I've been using dm-integrity with mdadm RAID on top, and LVM on top of that. Once it's all initialized an working, it works great. The biggest downsides I can think of are: 1. You have to do an initial initial initialization of checksums on the disk, which takes forever. 2. Automatically bringing up the dm-integrity layer of the plain disks at boot time is a bit of a dark art involving udev rules. If you skip the checksum initialization step and add the device to the RAID immediately, it _mostly_ works, because disk blocks are rewritten by the resync/reshape, but then on reboot it fails because some part of the system tries to read an otherwise unused sector (probably to find metadata or a partition signature of some kind?), and then determines the device is unusable because of read errors (dm-integrity errors show up as read failures to the next layer in the stack).
- Scramblejams 5y ago
  Thanks for sharing your experience. Do you run it in bitmap mode? If not, how’s the perf hit feel (I’m using spinners)? Does it have any ability to correct errors, or does it just cause the drive to drop when they appear? (I hoped it would write the corrected value back during an mdadm array check.) Does it reduce drive capacity at all? If not, I’m thinking maybe I could convert my array to it, one drive at a time, verrrry slowly.
  - martijnvds 5y ago
    I'm running in bitmap mode, on spinning drives. It reduces the drive capacity, but not by a noticable amount. I've set up my RAID to not haeve a "bad block list" so I _think_ it will try fixing it a few times, and drop the drive if there are too many errors in a row. Should be easy enough to try with a "test" RAID made out of a bunch of files (or even USB sticks).

martijnvds 5y ago

I've been using dm-integrity with mdadm RAID on top, and LVM on top of that. Once it's all initialized an working, it works great. The biggest downsides I can think of are: 1. You have to do an initial initial initialization of checksums on the disk, which takes forever. 2. Automatically bringing up the dm-integrity layer of the plain disks at boot time is a bit of a dark art involving udev rules. If you skip the checksum initialization step and add the device to the RAID immediately, it _mostly_ works, because disk blocks are rewritten by the resync/reshape, but then on reboot it fails because some part of the system tries to read an otherwise unused sector (probably to find metadata or a partition signature of some kind?), and then determines the device is unusable because of read errors (dm-integrity errors show up as read failures to the next layer in the stack).

Scramblejams 5y ago
Thanks for sharing your experience. Do you run it in bitmap mode? If not, how’s the perf hit feel (I’m using spinners)? Does it have any ability to correct errors, or does it just cause the drive to drop when they appear? (I hoped it would write the corrected value back during an mdadm array check.) Does it reduce drive capacity at all? If not, I’m thinking maybe I could convert my array to it, one drive at a time, verrrry slowly.
- martijnvds 5y ago
  I'm running in bitmap mode, on spinning drives. It reduces the drive capacity, but not by a noticable amount. I've set up my RAID to not haeve a "bad block list" so I _think_ it will try fixing it a few times, and drop the drive if there are too many errors in a row. Should be easy enough to try with a "test" RAID made out of a bunch of files (or even USB sticks).