3 ms·
How about block level checksums? Can LVM do that as well? Or is there some other way? ZFS dedup takes 320 bytes of RAM per unique block "record". So 1TB of RAM
by vardump 9y ago
How about block level checksums? Can LVM do that as well? Or is there some other way?
ZFS dedup takes 320 bytes of RAM per unique block "record". So 1TB of RAM is enough to dedup only a bit over 6 TB worth of unique blocks, when using a block size that works well with virtual machines — 4 kB.
One can of course use larger ZFS record size than 4kB. But virtual machine dedup savings drop very sharply as a result if record size does not match virtual machine filesystem block size and alignment. This happens, because there are exponentially more different combinations how 4 kB blocks can be arranged inside a bigger record size.
It's painful to format all virtual machine images to use say 64/128 kB filesystem block size to be able to efficiently use larger ZFS record size dedup.
I understand VDO dedup requires significantly less memory and uses only 4 kB blocks for dedup. This is ideal for VM storage application.
- tmd83 9y ago320 bytes? Doesn't that seem excessive or am I missing something.
- AstralStorm 9y agoHow about taking block level checksum a few levels higher and finally build a filesystem that has a Merkle tree checksum?
- anonuser123456 9y agoYou mean like ZFS?
- tremon 9y agoHow about block level checksums? Can LVM do that as well? Or is there some other way? LVM by itself doesn't do anything, it's a management layer on top of device-mapper. But yes, a block-level checksum exists since cryptsetup v2.0 (very recent), called dm-integrity: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/device-mapper/dm-integrity.txt https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...