7 ms·

A look at VDO, the new Linux compression layer

146 points by lima 9y ago

lima 9y ago
This is great news! At my employer, we're currently using btrfs in production for deduplication. It's relatively stable nowadays, but fragmentation and maintenance are massive issues for our use case.
- lunchables 9y ago
  On Redhat? Are you aware that Redhat has deprecated btrfs? I believe btrfs still used heavily in SUSE.
  - ioayman 9y ago
    Exactly! _" The Btrfs file system has been in Technology Preview state since the initial release of Red Hat Enterprise Linux 6. Red Hat will not be moving Btrfs to a fully supported feature and it will be removed in a future major release of Red Hat Enterprise Linux. The Btrfs file system did receive numerous updates from the upstream in Red Hat Enterprise Linux 7.4 and will remain available in the Red Hat Enterprise Linux 7 series. However, this is the last planned update to this feature."_ Source: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.4_release_notes/chap-red_hat_enterprise_linux-7.4_release_notes-deprecated_functionality https://access.redhat.com/documentation/en-us/red_hat_enterp...
  - lima 9y ago
    Yes, we build and test our own kernels and userland tools and figured out which kernel versions work and which ones don't. You've got to keep a close eye on the IRC channel and mailing lists. We never had any data loss but a lot of unplanned downtime until it got sufficiently stable - for our use case - a few months ago. Works well but we'd rather not have to worry about these kind of things. (even before it was deprecated, it was marked a "technology preview", rightfully so - if you want to use btrfs, you need to stay very close to mainline)
- deleted 9y ago
  [deleted]
vardump 9y ago
I wonder if you can run VDO on top of ZFS. Or is there a better way to handle snapshots and data integrity (block level checksums)? Without block level checksums, a single corrupted data block could corrupt half of your virtual machine images...
- dmm 9y ago
  LVM provides CoW snapshots. But why would you use this instead of ZFS's native compression or dedup ?
  - vardump 9y ago
    How about block level checksums? Can LVM do that as well? Or is there some other way? ZFS dedup takes 320 bytes of RAM per unique block "record". So 1TB of RAM is enough to dedup only a bit over 6 TB worth of unique blocks, when using a block size that works well with virtual machines — 4 kB. One can of course use larger ZFS record size than 4kB. But virtual machine dedup savings drop very sharply as a result if record size does not match virtual machine filesystem block size and alignment. This happens, because there are exponentially more different combinations how 4 kB blocks can be arranged inside a bigger record size. It's painful to format all virtual machine images to use say 64/128 kB filesystem block size to be able to efficiently use larger ZFS record size dedup. I understand VDO dedup requires significantly less memory and uses only 4 kB blocks for dedup. This is ideal for VM storage application.
    - tmd83 9y ago
      320 bytes? Doesn't that seem excessive or am I missing something.
    - AstralStorm 9y ago
      How about taking block level checksum a few levels higher and finally build a filesystem that has a Merkle tree checksum?
      - anonuser123456 9y ago
faragon 9y ago
For data compression it uses the LZ4 format [1] (real-time LZ77-like data compressor -without entropy encoding, just string references and literals-, with small blocks so its LUT-based O(1) search is always in the data cache). [1] https://rhelblog.redhat.com/2018/02/05/understanding-the-concepts-behind-virtual-data-optimizer-vdo-in-rhel-7-5-beta/ https://rhelblog.redhat.com/2018/02/05/understanding-the-con...
jimmies 9y ago
If you plan to use it for your personal traditional storage HDD, you have to keep in mind that it will render most (if not all) data recovery tools naught. Tools such as PhotoRec rely on the file headers to determine the file type to recover the data. I have a long-term storage HDD that I thought I would never need data recovery on it until I fucked it up good by a wrong partitioning command. Normally those types of mistakes can be recovered by TestDisk but not that time. I realized my only hope was to use PhotoRec to recover what I can from the garbage that I overwrote on the disk. Thanks to PhotoRec I could recover most of them with messed up names. I was so thankful I didn't use any fancy compression algorithm. Tools such as LVM, SSD backing storage, btrfs, compression and stuff are all nice when you understand their limitations. For now, I don't use all of my storage so I just create ext3, ext4, and exFAT partitions to store my data for the maximal chance of recovery, whether it is due to hardware or software.
- zokier 9y ago
  On the other hand if you are using full-disk encryption then you will be fucked anyways.
  - Arbalest 9y ago
    Depends. If you mean soft compression (not ATA level crypto) you may be able to back up the header externally and still decrypt it. The data will be scrambled where it would have been scrambled anyway, and unmodified regions will remain unmodified.
- vardump 9y ago
  Using something like VDO for personal storage would be pretty pointless. Unless, of course, you have a ton of similar virtual machines or other data with high amount of 4 kB block size aligned redundancy.
  - globalc 9y ago
    If your data directories have much redundancy (so deduplication can help), or are nicely compressable, then VDO can also make sense for home setups. Like companies, you would consider things like "does it make sense for my kind of data", "do I have enough CPU cycles/ram to spend on getting storage down to some degree" and "am I ok with the additional layer, so more complicated setup".
d33 9y ago
At first I was confused because I thought that someone miraculously implemented compression with no context switching via VDSO. Funny to see how wrong I was ;) Could someone explain how VDO works to me? Based on the example, it looks like another DM backend, like LUKS and such. It exposes a virtual device backed by a real one, adding a layer of compression, am I reading it right? I can also see that we're specifying a "logical size" that is exposed to the user. How much space is really used and how is it allocated? Can I grow the logical size later? Apart from this - what is the status of the patch, is it redhat-only or is it on its way to the upstream?
- chr15p 9y ago
  > Apart from this - what is the status of the patch, is it redhat-only or is it on its way to the upstream? I had a conversation with some of the RH kernel devs a few weeks ago and from my (hazy) memory, this comes from Red Hats acquisition of permabit so it was closed source, its now GPL'd but needs a fair amount of tidying up and bits rewriting to get it into a state that the kernel developers would accept it (they reimplemented some standard kernel features so they didn't have to link to GPL'd code). So it will be upstreamed but it will take time. In the meantime its released for RHEL only, not even in Fedora. Iirc could be ported to other distros (its 100% GPLed) its just not been.
- sweettea 9y ago
  > It exposes a virtual device backed by a real one, adding a layer of compression, am I reading it right? And deduplication, yes. >Can I grow the logical size later? Yes; the man page for the program discusses the growLogical subcommand briefly: https://github.com/dm-vdo/vdo/blob/6.1.1.24/vdo-manager/man/vdo.8#L281 https://github.com/dm-vdo/vdo/blob/6.1.1.24/vdo-manager/man/... >I can also see that we're specifying a "logical size" that is exposed to the user. How much space is really used and how is it allocated? There's a vdostats command mentioned in the article that exposes how much actual space is available and used. >Apart from this - what is the status of the patch, is it redhat-only or is it on its way to the upstream? It has not yet been submitted as a kernel patch.
  - lotyrin 9y ago
shmerl 9y ago
So it's a layer under the filesystem and should work with any filesystem in theory? So would XFS + VDO be better than let's say using compression in BTRFS?
- lunchables 9y ago
  I'm assuming so, because Redhat is getting rid of btrfs. That would explain why they announced the two in the same release notes.
- globalc 9y ago
  Yes, VDO is transparent for the layers above, like filesystems. XFS+VDO might be preferable to BTRFS, if it has the features you need. - LVM could be used for snapshots, but the snapshots from BTRFS might be preferable in corner cases where many snapshots are done (snapshots in XFS are being worked on AFAIK, these could then be comparable) - XFS+VDO combination might be more reliable, it's a completely supported combination on RHEL, whereas BTRFS is just techpreview - only missing feature coming to mind is checksums over data blocks, XFS is only providing checksums over meta data
  - LeoPanthera 9y ago
    How do filesystems cope with the apparent capacity of the disk changing? If you put a lot of data on that compresses really well, suddenly your disk will appear to have become larger.
    - globalc 9y ago
      The capacity is not changing 'rapidly', instead the whole VDO device is thin provisioned. Let's assume a 1TB harddisk, then when creating a VDO device ontop, you can have that directly appear as 3TB, so 'thin provisioned'. The filesystem ontop always sees 3TB, unless you explicitly modify the VDO device. Of course, you have to monitor the VDO status tightly: if you happen to store data on the filesystem which is absolutely unique, has no zeros and is uncompressable, then dedup/compression can not do anything. The VDO device can only consume up to ~1TB of such data. Your monitoring should detect the low space on the VDO backend before, and you should then either stop writing or extend the backend device.
zokier 9y ago
Considering that it is based on blazingly fast lz4, using twice as much time as non-VDO in this simple test seems ..bad? I suppose the dedup stage is really expensive here. On a setup like this with simple compression, I would have expected the results to be inverted.
- rurban 9y ago
  I would have thought they would consider zstd also, because that would be even better and easier tunable. But I guess there are some license problems still.
  - zokier 9y ago
    VDO probably predates zstd (1.0) release fair bit.
  - 0xcde4c3db 9y ago
    Reading some blog posts and such about ZFS compression, I got the impression that simpler algorithms such as LZ4 and LZO are typically preferred for transparent storage compression. Presumably a balance must be struck between the benefits of writing less data and the cost of taking CPU time away from other code.
mindslight 9y ago
Remember when the N in O(N) was the size of the problem, rather than the quantity of O(lg N') indirection layers? Pepperidge Ph4rm remembers.
jlarocco 9y ago
The opening question made me laugh because I actually do feel I have too much storage. I haven't bought a new hard drive in years, and I'm only using maybe 1.5 Tb of ~3 Tb, and most of that space is used up by RAW images from my DSLR. At the rate I'm going, it'll be 2 or 3 years before I need a new drive.
- sweettea 9y ago
  Personally I do agree about hard drives. They're so enormous these days that I'll never use even a moderately sized one. I'm spoiled, though, and love my SSDs; they're fast, smaller, and more expensive, and just a couple of VMs quickly eat up a bunch of expensive space. With VDO, I personally run 20 VMs instead of 8.

146 points by lima 9y ago

lima 9y ago

This is great news! At my employer, we're currently using btrfs in production for deduplication. It's relatively stable nowadays, but fragmentation and maintenance are massive issues for our use case.

lunchables 9y ago
On Redhat? Are you aware that Redhat has deprecated btrfs? I believe btrfs still used heavily in SUSE.
- ioayman 9y ago
  Exactly! _" The Btrfs file system has been in Technology Preview state since the initial release of Red Hat Enterprise Linux 6. Red Hat will not be moving Btrfs to a fully supported feature and it will be removed in a future major release of Red Hat Enterprise Linux. The Btrfs file system did receive numerous updates from the upstream in Red Hat Enterprise Linux 7.4 and will remain available in the Red Hat Enterprise Linux 7 series. However, this is the last planned update to this feature."_ Source: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.4_release_notes/chap-red_hat_enterprise_linux-7.4_release_notes-deprecated_functionality https://access.redhat.com/documentation/en-us/red_hat_enterp...
- lima 9y ago
  Yes, we build and test our own kernels and userland tools and figured out which kernel versions work and which ones don't. You've got to keep a close eye on the IRC channel and mailing lists. We never had any data loss but a lot of unplanned downtime until it got sufficiently stable - for our use case - a few months ago. Works well but we'd rather not have to worry about these kind of things. (even before it was deprecated, it was marked a "technology preview", rightfully so - if you want to use btrfs, you need to stay very close to mainline)
deleted 9y ago
[deleted]

vardump 9y ago

I wonder if you can run VDO on top of ZFS. Or is there a better way to handle snapshots and data integrity (block level checksums)? Without block level checksums, a single corrupted data block could corrupt half of your virtual machine images...

dmm 9y ago
LVM provides CoW snapshots. But why would you use this instead of ZFS's native compression or dedup ?
- vardump 9y ago
  How about block level checksums? Can LVM do that as well? Or is there some other way? ZFS dedup takes 320 bytes of RAM per unique block "record". So 1TB of RAM is enough to dedup only a bit over 6 TB worth of unique blocks, when using a block size that works well with virtual machines — 4 kB. One can of course use larger ZFS record size than 4kB. But virtual machine dedup savings drop very sharply as a result if record size does not match virtual machine filesystem block size and alignment. This happens, because there are exponentially more different combinations how 4 kB blocks can be arranged inside a bigger record size. It's painful to format all virtual machine images to use say 64/128 kB filesystem block size to be able to efficiently use larger ZFS record size dedup. I understand VDO dedup requires significantly less memory and uses only 4 kB blocks for dedup. This is ideal for VM storage application.
  - tmd83 9y ago
    320 bytes? Doesn't that seem excessive or am I missing something.
  - AstralStorm 9y ago
    How about taking block level checksum a few levels higher and finally build a filesystem that has a Merkle tree checksum?
    - anonuser123456 9y ago

faragon 9y ago

For data compression it uses the LZ4 format [1] (real-time LZ77-like data compressor -without entropy encoding, just string references and literals-, with small blocks so its LUT-based O(1) search is always in the data cache). [1] https://rhelblog.redhat.com/2018/02/05/understanding-the-concepts-behind-virtual-data-optimizer-vdo-in-rhel-7-5-beta/ https://rhelblog.redhat.com/2018/02/05/understanding-the-con...

jimmies 9y ago

If you plan to use it for your personal traditional storage HDD, you have to keep in mind that it will render most (if not all) data recovery tools naught. Tools such as PhotoRec rely on the file headers to determine the file type to recover the data. I have a long-term storage HDD that I thought I would never need data recovery on it until I fucked it up good by a wrong partitioning command. Normally those types of mistakes can be recovered by TestDisk but not that time. I realized my only hope was to use PhotoRec to recover what I can from the garbage that I overwrote on the disk. Thanks to PhotoRec I could recover most of them with messed up names. I was so thankful I didn't use any fancy compression algorithm. Tools such as LVM, SSD backing storage, btrfs, compression and stuff are all nice when you understand their limitations. For now, I don't use all of my storage so I just create ext3, ext4, and exFAT partitions to store my data for the maximal chance of recovery, whether it is due to hardware or software.

zokier 9y ago
On the other hand if you are using full-disk encryption then you will be fucked anyways.
- Arbalest 9y ago
  Depends. If you mean soft compression (not ATA level crypto) you may be able to back up the header externally and still decrypt it. The data will be scrambled where it would have been scrambled anyway, and unmodified regions will remain unmodified.
vardump 9y ago
Using something like VDO for personal storage would be pretty pointless. Unless, of course, you have a ton of similar virtual machines or other data with high amount of 4 kB block size aligned redundancy.
- globalc 9y ago
  If your data directories have much redundancy (so deduplication can help), or are nicely compressable, then VDO can also make sense for home setups. Like companies, you would consider things like "does it make sense for my kind of data", "do I have enough CPU cycles/ram to spend on getting storage down to some degree" and "am I ok with the additional layer, so more complicated setup".

d33 9y ago

At first I was confused because I thought that someone miraculously implemented compression with no context switching via VDSO. Funny to see how wrong I was ;) Could someone explain how VDO works to me? Based on the example, it looks like another DM backend, like LUKS and such. It exposes a virtual device backed by a real one, adding a layer of compression, am I reading it right? I can also see that we're specifying a "logical size" that is exposed to the user. How much space is really used and how is it allocated? Can I grow the logical size later? Apart from this - what is the status of the patch, is it redhat-only or is it on its way to the upstream?

chr15p 9y ago
> Apart from this - what is the status of the patch, is it redhat-only or is it on its way to the upstream? I had a conversation with some of the RH kernel devs a few weeks ago and from my (hazy) memory, this comes from Red Hats acquisition of permabit so it was closed source, its now GPL'd but needs a fair amount of tidying up and bits rewriting to get it into a state that the kernel developers would accept it (they reimplemented some standard kernel features so they didn't have to link to GPL'd code). So it will be upstreamed but it will take time. In the meantime its released for RHEL only, not even in Fedora. Iirc could be ported to other distros (its 100% GPLed) its just not been.
sweettea 9y ago
> It exposes a virtual device backed by a real one, adding a layer of compression, am I reading it right? And deduplication, yes. >Can I grow the logical size later? Yes; the man page for the program discusses the growLogical subcommand briefly: https://github.com/dm-vdo/vdo/blob/6.1.1.24/vdo-manager/man/vdo.8#L281 https://github.com/dm-vdo/vdo/blob/6.1.1.24/vdo-manager/man/... >I can also see that we're specifying a "logical size" that is exposed to the user. How much space is really used and how is it allocated? There's a vdostats command mentioned in the article that exposes how much actual space is available and used. >Apart from this - what is the status of the patch, is it redhat-only or is it on its way to the upstream? It has not yet been submitted as a kernel patch.
- lotyrin 9y ago

shmerl 9y ago

So it's a layer under the filesystem and should work with any filesystem in theory? So would XFS + VDO be better than let's say using compression in BTRFS?

lunchables 9y ago
I'm assuming so, because Redhat is getting rid of btrfs. That would explain why they announced the two in the same release notes.
globalc 9y ago
Yes, VDO is transparent for the layers above, like filesystems. XFS+VDO might be preferable to BTRFS, if it has the features you need. - LVM could be used for snapshots, but the snapshots from BTRFS might be preferable in corner cases where many snapshots are done (snapshots in XFS are being worked on AFAIK, these could then be comparable) - XFS+VDO combination might be more reliable, it's a completely supported combination on RHEL, whereas BTRFS is just techpreview - only missing feature coming to mind is checksums over data blocks, XFS is only providing checksums over meta data
- LeoPanthera 9y ago
  How do filesystems cope with the apparent capacity of the disk changing? If you put a lot of data on that compresses really well, suddenly your disk will appear to have become larger.
  - globalc 9y ago
    The capacity is not changing 'rapidly', instead the whole VDO device is thin provisioned. Let's assume a 1TB harddisk, then when creating a VDO device ontop, you can have that directly appear as 3TB, so 'thin provisioned'. The filesystem ontop always sees 3TB, unless you explicitly modify the VDO device. Of course, you have to monitor the VDO status tightly: if you happen to store data on the filesystem which is absolutely unique, has no zeros and is uncompressable, then dedup/compression can not do anything. The VDO device can only consume up to ~1TB of such data. Your monitoring should detect the low space on the VDO backend before, and you should then either stop writing or extend the backend device.

zokier 9y ago

Considering that it is based on blazingly fast lz4, using twice as much time as non-VDO in this simple test seems ..bad? I suppose the dedup stage is really expensive here. On a setup like this with simple compression, I would have expected the results to be inverted.

rurban 9y ago
I would have thought they would consider zstd also, because that would be even better and easier tunable. But I guess there are some license problems still.
- zokier 9y ago
  VDO probably predates zstd (1.0) release fair bit.
- 0xcde4c3db 9y ago
  Reading some blog posts and such about ZFS compression, I got the impression that simpler algorithms such as LZ4 and LZO are typically preferred for transparent storage compression. Presumably a balance must be struck between the benefits of writing less data and the cost of taking CPU time away from other code.

mindslight 9y ago

Remember when the N in O(N) was the size of the problem, rather than the quantity of O(lg N') indirection layers? Pepperidge Ph4rm remembers.

jlarocco 9y ago

The opening question made me laugh because I actually do feel I have too much storage. I haven't bought a new hard drive in years, and I'm only using maybe 1.5 Tb of ~3 Tb, and most of that space is used up by RAW images from my DSLR. At the rate I'm going, it'll be 2 or 3 years before I need a new drive.

sweettea 9y ago
Personally I do agree about hard drives. They're so enormous these days that I'll never use even a moderately sized one. I'm spoiled, though, and love my SSDs; they're fast, smaller, and more expensive, and just a couple of VMs quickly eat up a bunch of expensive space. With VDO, I personally run 20 VMs instead of 8.