4 ms·
LVM provides CoW snapshots. But why would you use this instead of ZFS's native compression or dedup ?
by dmm 9y ago
LVM provides CoW snapshots. But why would you use this instead of ZFS's native compression or dedup ?
- vardump 9y agoHow about block level checksums? Can LVM do that as well? Or is there some other way? ZFS dedup takes 320 bytes of RAM per unique block "record". So 1TB of RAM is enough to dedup only a bit over 6 TB worth of unique blocks, when using a block size that works well with virtual machines — 4 kB. One can of course use larger ZFS record size than 4kB. But virtual machine dedup savings drop very sharply as a result if record size does not match virtual machine filesystem block size and alignment. This happens, because there are exponentially more different combinations how 4 kB blocks can be arranged inside a bigger record size. It's painful to format all virtual machine images to use say 64/128 kB filesystem block size to be able to efficiently use larger ZFS record size dedup. I understand VDO dedup requires significantly less memory and uses only 4 kB blocks for dedup. This is ideal for VM storage application.
- tmd83 9y ago320 bytes? Doesn't that seem excessive or am I missing something.
- AstralStorm 9y agoHow about taking block level checksum a few levels higher and finally build a filesystem that has a Merkle tree checksum?
- anonuser123456 9y agoYou mean like ZFS?
- tremon 9y agoHow about block level checksums? Can LVM do that as well? Or is there some other way? LVM by itself doesn't do anything, it's a management layer on top of device-mapper. But yes, a block-level checksum exists since cryptsetup v2.0 (very recent), called dm-integrity: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/device-mapper/dm-integrity.txt https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- sweettea 9y agoZFS's dedupe is, I believe, 64k block dedupe; VDO's is 4k block dedupe.
- vardump 9y agoZFS dedup block size is whatever you set dataset record size to be. Powers of two from 0.5 kB to 256 kB are allowed at least. So you can set ZFS to dedup 4 kB blocks. It just requires a boatload of RAM.
- LeoPanthera 9y agoZFS’ dedup is essentially useless, since it requires that the entire dedup table is held in RAM. This is one area where btrfs has a huge advantage, dedup can be done offline, rather than in real time.
- ryao 9y agoZFS’ deduplication does not require the deduplication table be held in RAM. Hoewever, the performance is IOPS bound. The only way to improve on that is to keep it in RAM or have a very fast L2ARC. Many modifications also require quite a large number of writes, which also slows things down. My point is that the system will not stop working if the DDT is not in cache. It would still run, but it would not be fast.