6 ms·

I wonder if you can run VDO on top of ZFS. Or is there a better way to handle snapshots and data integrity (block level checksums)? Without block level checks

I wonder if you can run VDO on top of ZFS. Or is there a better way to handle snapshots and data integrity (block level checksums)? Without block level checksums, a single corrupted data block could corrupt half of your virtual machine images...

dmm 9y ago
LVM provides CoW snapshots. But why would you use this instead of ZFS's native compression or dedup ?
- vardump 9y ago
  How about block level checksums? Can LVM do that as well? Or is there some other way? ZFS dedup takes 320 bytes of RAM per unique block "record". So 1TB of RAM is enough to dedup only a bit over 6 TB worth of unique blocks, when using a block size that works well with virtual machines — 4 kB. One can of course use larger ZFS record size than 4kB. But virtual machine dedup savings drop very sharply as a result if record size does not match virtual machine filesystem block size and alignment. This happens, because there are exponentially more different combinations how 4 kB blocks can be arranged inside a bigger record size. It's painful to format all virtual machine images to use say 64/128 kB filesystem block size to be able to efficiently use larger ZFS record size dedup. I understand VDO dedup requires significantly less memory and uses only 4 kB blocks for dedup. This is ideal for VM storage application.
  - tmd83 9y ago
    320 bytes? Doesn't that seem excessive or am I missing something.
  - AstralStorm 9y ago
    How about taking block level checksum a few levels higher and finally build a filesystem that has a Merkle tree checksum?
    - anonuser123456 9y ago
      You mean like ZFS?
  - tremon 9y ago
    How about block level checksums? Can LVM do that as well? Or is there some other way? LVM by itself doesn't do anything, it's a management layer on top of device-mapper. But yes, a block-level checksum exists since cryptsetup v2.0 (very recent), called dm-integrity: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/device-mapper/dm-integrity.txt https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- sweettea 9y ago
  ZFS's dedupe is, I believe, 64k block dedupe; VDO's is 4k block dedupe.
  - vardump 9y ago
    ZFS dedup block size is whatever you set dataset record size to be. Powers of two from 0.5 kB to 256 kB are allowed at least. So you can set ZFS to dedup 4 kB blocks. It just requires a boatload of RAM.
- LeoPanthera 9y ago
  ZFS’ dedup is essentially useless, since it requires that the entire dedup table is held in RAM. This is one area where btrfs has a huge advantage, dedup can be done offline, rather than in real time.
  - ryao 9y ago
    ZFS’ deduplication does not require the deduplication table be held in RAM. Hoewever, the performance is IOPS bound. The only way to improve on that is to keep it in RAM or have a very fast L2ARC. Many modifications also require quite a large number of writes, which also slows things down. My point is that the system will not stop working if the DDT is not in cache. It would still run, but it would not be fast.
zzzcpan 9y ago
Compression usually has checksumming already.
- vardump 9y ago
  Compression and checksumming have nothing to do with each other. I can't think of even one compression algorithm that implements checksums. That's why archive formats (like ZIP, 7z, etc.) need to implement checksums separately — inflate and deflate algorithms ZIP uses don't have any kind of built in checksums.
  - zzzcpan 9y ago
    Applications use libraries that implement data compression. Such libraries usually have checksumming in their algorithms, because compression is not very practical without it. LZ4, that VDO uses, has checksumming too.
    - vardump 9y ago
      LZ4 frame format (called LZ4F) has checksums — a stream that contains LZ4 compressed data. But LZ4 algorithm itself doesn't have any checksums. A block device layer would use the "raw" algorithm, not any frame/container format and use something like SHA256 for checksums. Take a look at LZ4 source: https://github.com/lz4/lz4/tree/dev/lib https://github.com/lz4/lz4/tree/dev/lib VDO LZ4 source: https://github.com/dm-vdo/kvdo/blob/master/vdo/base/lz4.c https://github.com/dm-vdo/kvdo/blob/master/vdo/base/lz4.c No LZ4F frame format or checksum in sight.
      - zzzcpan 9y ago
        Yes, you are right, they use raw lz4 and don't do checksumming there. And I can only find checksumming in superblock code and volume geometry code in vdo repository. So it doesn't look like they do it on regular data blocks at all, probably just assume you have a properly working FTL. (by the way, sha256 is a slow cryptographic hash, not a checksumming one, like crc32)
xxpor 9y ago
FreeNAS supports lz4 compression on ZFS, but I'm not sure if that a ZFS feature or what.
- vardump 9y ago
  LZ4 is a ZFS feature as far as I know, and works really well. So well, that I think it should be default. But dedup is why I'm interested in VDO.
- __jal 9y ago
  It is ZFS, and is on by default. ZFS also offers gzip compression, and you can "turn up" the LZ4 from the default if you wish. I've almost always left it at the default; on my home file server (slightly long-in-the-tooth 6 core xeon, enterprise-grade spinning rust), throughput is noticeably faster on compressible data. (ZFS dedupe should only be considered for weird cases, like if you somehow have a ton of RAM but very limited storage. Frankly, at this point I think it is an attractive nuisance that leads beginners down a dangerous path and should be removed, or at least the commands to enable it should be given loud, scary confirmation messages.)
globalc 9y ago
Nothing prevents you from running VDO ontop of ZFS, apart of that not being in focus/tested at Red Hat. I do not expect issues from purely running both layers, but maybe things like proper mounting/umounting via fstab might not be in place. If ZFS gives you block level checksums, then you could use compression/dedup from VDO. Just activating i.e. compression on both layers would be waste of cycles.
MawKKe 9y ago
For block level checksums there is dm-integrity :) It's dm layers all the way down...
- ryao 9y ago
  That writes all data twice. If it really does emulate hard drives that have per sector tags, it should have the same weakness. That being a misdirected write will put data with a correct checksum in the wrong location, which will be served without knowing. It’s documentation suggests that it can detect on journal replay whether the data was written where it was supposed to be, but it is not going to catch sites clobbered by a misdirected write every time because sone times the wrong sector is perfectly overwritten with no overlap with other sectors. This is not a replacement for ZFS zvold, which do checksum each block.
ryao 9y ago
It should work.

dmm 9y ago

LVM provides CoW snapshots. But why would you use this instead of ZFS's native compression or dedup ?

vardump 9y ago
How about block level checksums? Can LVM do that as well? Or is there some other way? ZFS dedup takes 320 bytes of RAM per unique block "record". So 1TB of RAM is enough to dedup only a bit over 6 TB worth of unique blocks, when using a block size that works well with virtual machines — 4 kB. One can of course use larger ZFS record size than 4kB. But virtual machine dedup savings drop very sharply as a result if record size does not match virtual machine filesystem block size and alignment. This happens, because there are exponentially more different combinations how 4 kB blocks can be arranged inside a bigger record size. It's painful to format all virtual machine images to use say 64/128 kB filesystem block size to be able to efficiently use larger ZFS record size dedup. I understand VDO dedup requires significantly less memory and uses only 4 kB blocks for dedup. This is ideal for VM storage application.
- tmd83 9y ago
  320 bytes? Doesn't that seem excessive or am I missing something.
- AstralStorm 9y ago
  How about taking block level checksum a few levels higher and finally build a filesystem that has a Merkle tree checksum?
  - anonuser123456 9y ago
    You mean like ZFS?
- tremon 9y ago
  How about block level checksums? Can LVM do that as well? Or is there some other way? LVM by itself doesn't do anything, it's a management layer on top of device-mapper. But yes, a block-level checksum exists since cryptsetup v2.0 (very recent), called dm-integrity: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/device-mapper/dm-integrity.txt https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
sweettea 9y ago
ZFS's dedupe is, I believe, 64k block dedupe; VDO's is 4k block dedupe.
- vardump 9y ago
  ZFS dedup block size is whatever you set dataset record size to be. Powers of two from 0.5 kB to 256 kB are allowed at least. So you can set ZFS to dedup 4 kB blocks. It just requires a boatload of RAM.
LeoPanthera 9y ago
ZFS’ dedup is essentially useless, since it requires that the entire dedup table is held in RAM. This is one area where btrfs has a huge advantage, dedup can be done offline, rather than in real time.
- ryao 9y ago
  ZFS’ deduplication does not require the deduplication table be held in RAM. Hoewever, the performance is IOPS bound. The only way to improve on that is to keep it in RAM or have a very fast L2ARC. Many modifications also require quite a large number of writes, which also slows things down. My point is that the system will not stop working if the DDT is not in cache. It would still run, but it would not be fast.

zzzcpan 9y ago

Compression usually has checksumming already.

vardump 9y ago
Compression and checksumming have nothing to do with each other. I can't think of even one compression algorithm that implements checksums. That's why archive formats (like ZIP, 7z, etc.) need to implement checksums separately — inflate and deflate algorithms ZIP uses don't have any kind of built in checksums.
- zzzcpan 9y ago
  Applications use libraries that implement data compression. Such libraries usually have checksumming in their algorithms, because compression is not very practical without it. LZ4, that VDO uses, has checksumming too.
  - vardump 9y ago
    LZ4 frame format (called LZ4F) has checksums — a stream that contains LZ4 compressed data. But LZ4 algorithm itself doesn't have any checksums. A block device layer would use the "raw" algorithm, not any frame/container format and use something like SHA256 for checksums. Take a look at LZ4 source: https://github.com/lz4/lz4/tree/dev/lib https://github.com/lz4/lz4/tree/dev/lib VDO LZ4 source: https://github.com/dm-vdo/kvdo/blob/master/vdo/base/lz4.c https://github.com/dm-vdo/kvdo/blob/master/vdo/base/lz4.c No LZ4F frame format or checksum in sight.
    - zzzcpan 9y ago
      Yes, you are right, they use raw lz4 and don't do checksumming there. And I can only find checksumming in superblock code and volume geometry code in vdo repository. So it doesn't look like they do it on regular data blocks at all, probably just assume you have a properly working FTL. (by the way, sha256 is a slow cryptographic hash, not a checksumming one, like crc32)

xxpor 9y ago

FreeNAS supports lz4 compression on ZFS, but I'm not sure if that a ZFS feature or what.

vardump 9y ago
LZ4 is a ZFS feature as far as I know, and works really well. So well, that I think it should be default. But dedup is why I'm interested in VDO.
__jal 9y ago
It is ZFS, and is on by default. ZFS also offers gzip compression, and you can "turn up" the LZ4 from the default if you wish. I've almost always left it at the default; on my home file server (slightly long-in-the-tooth 6 core xeon, enterprise-grade spinning rust), throughput is noticeably faster on compressible data. (ZFS dedupe should only be considered for weird cases, like if you somehow have a ton of RAM but very limited storage. Frankly, at this point I think it is an attractive nuisance that leads beginners down a dangerous path and should be removed, or at least the commands to enable it should be given loud, scary confirmation messages.)

globalc 9y ago

Nothing prevents you from running VDO ontop of ZFS, apart of that not being in focus/tested at Red Hat. I do not expect issues from purely running both layers, but maybe things like proper mounting/umounting via fstab might not be in place. If ZFS gives you block level checksums, then you could use compression/dedup from VDO. Just activating i.e. compression on both layers would be waste of cycles.

MawKKe 9y ago

For block level checksums there is dm-integrity :) It's dm layers all the way down...

ryao 9y ago
That writes all data twice. If it really does emulate hard drives that have per sector tags, it should have the same weakness. That being a misdirected write will put data with a correct checksum in the wrong location, which will be served without knowing. It’s documentation suggests that it can detect on journal replay whether the data was written where it was supposed to be, but it is not going to catch sites clobbered by a misdirected write every time because sone times the wrong sector is perfectly overwritten with no overlap with other sectors. This is not a replacement for ZFS zvold, which do checksum each block.

ryao 9y ago

It should work.