4 ms·
"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and spec
by UltraSane 2y ago
"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads."
This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compression works really well on syslog servers where I've seen 6:1 savings.
The effectiveness of dedupe is strongly affected by the size of the blocks being hashed, with the smaller the better. As the blocks get smaller the odds of having a matching block grow rapidly. In my experience 4KB is my preferred block size.
- acdha 2y ago> Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. Don’t you compress these directly? I normally see at least twice that for logs doing it at the process level.
- UltraSane 2y agoWhat software?
- SteveNuts 2y agoLogrotate
- acdha 2y agoLog rotate, cron, or simply having something like Varnish or Apache log to a pipe which is something like bzip2 or zstd. The main question is whether you want to easily access the current stream - e.g. I had uncompressed logs being forwarded to CloudWatch so I had daemons logging to timestamped files with a post-rotate compression command which would run after the last write.
- UltraSane 2y agoThat is one wrinkle of using storage based dedupe/compression is you need to avoid doing compression on the client to avoid compressing already compressed data. When a company I worked at first got their Pure array they were using windows file compression heavily and had to disable it as the storage array was now doing it automatically.
- acdha 2y agoDefinitely. We love building abstraction layers but at some point you really need to make decisions across the entire stack.
- chasil 2y agoLogrotate is the rhel utility, likely present in Fedora, that is easily adapted for custom log handling. I still have rhel5 and I use it there. CentOS made it famous. I don't know if it has a foothold in the Debian family.
- E39M5S62 2y agologrotate is used on Debian and plenty of other distros. It seems pretty widely used, though maybe not as much so now that things log through systemd.
- pezezin 2y agoYes, that ratio is very small. I built a very simple, custom syslog solution, a syslog-ng server writing directly to a TimescaleDB hypertable (https://www.timescale.com/ https://www.timescale.com/) that is then presented as a Grafana dashboard, and I am getting a 30x compression ratio.
- pdimitar 2y agoWould love to see your solution -- if it's open source.
- deleted 2y ago[deleted]
- pezezin 2y agoSure, no problem. But after you asked me, I realized that the system was not properly documented anywhere, I didn't even have a repo with the configuration files, what an embarrassment :( I just created the repo and uploaded the documentation, please give me some more time to write the documentation: https://github.com/josefrcm/simple-syslog-service https://github.com/josefrcm/simple-syslog-service
- jorvi 2y ago> In my experience 4KB is my preferred block size That makes sense considering Advanced Format harddrives already have a 4K physical sector size, and if you properly low-level format them (to get rid of the ridiculous Windows XP compatibility) they also have 4K logical sector size. I imagine there might be some real performance benefits to having all of those match up.
- UltraSane 2y agoIn the early days of VMware people had a lot of VMs that were converted from physical machines and this causes a nasty alignment issue between the VMDK blocks and the blocks on your storage array. The effect was to always add one block to every read operation, and in the worst case of reading one block would double the load on the storage array. On NetApp this could only be fixed when the VM wasn't running.
- Joe_Cool 2y agoEven with the rudimentary Dedup features of NTFS on a Windows Hyper-V Server all running the same base image I can overprovision the 512GB partition to almost 2 GB. You need to be careful and do staggered updates in the VMs or it'll spectacularly explode but it's possible and quite performant for less than mission critical VMs.
- tw04 2y agoI think you mean 2TB volume? But yes, this works. But also: if you're doing anything production, I'd strongly recommend doing deduplication on the back-end storage array, not at the NTFS layer. It'll be more performant and almost assuredly have better space savings.
- Joe_Cool 2y agoFor sure it's not for production. At least not for stuff that's critical. MS also doesn't recommend using it for live VHDX. The partition/NTFS volume is 512GB. It currently stores 1.3 TB of "dedupped" data and has about 200GB free. Dedup runs asynchronously in the background and as a job during off hours. It's a typo, yes. Thanks.
- wongarsu 2y agoI haven't tried it myself, but the widely quoted number for old ZFS dedup is that you need 5GB of RAM for every 1TB of disk space. Considering that 1 TB of disk space currently costs about $15 and 5GB of server RAM about $25, you need a 3:1 dedupe ratio just to break even. If your data is a good fit you might get away with 1GB per TB, but if you are out of luck the 5GB might not even be enough. That's why the article speaks of ZFS dedup having a small sweet spot that your data has to hit, and why most people don't bother Other file systems tend to prefer offline dedupe which has more favorable economics
- UltraSane 2y agoWhy does it need so much RAM? It should only need to store the block hashes which should not need anywhere near that much RAM. Inline dedupe is pretty much standard on high-end storage arrays nowadays.
- remexre 2y ago(5GiB / 1TiB) * 4KiB to bits ((5 gibibytes) / (1 tebibyte)) × (4 kibibytes) = 160 bits
- AndrewDavis 2y agoThe linked blog post covers this, and the improvements made to make the new dedup better.
- floating-io 2y agoThat doesn't account for OpEx, though, such as power...
- wongarsu 2y agoAssuming something reasonable like 20TB Toshiba MG10 HDDs and 64GB DDR4 ECC RAM, quick googling suggests that 1TB of disk space uses about 0.2-0.4W of power (0.2 in idle, 0.4 while writing), 5GB of RAM about 0.3-0.5W. So your break even on power is a bit earlier depending on the access pattern, but in the same ball park.
- wmf 2y agoVMs are known to benefit from dedupe so yes, you'll see benefits there. ZFS is a general-purpose filesystem not just an enterprise SAN so many ZFS users aren't running VMs. Dedupe/compression works really well on syslog I apologize for the pedantry but dedupe and compression aren't the same thing (although they tend to be bundled in the enterprise storage world). Logs are probably benefiting from compression not dedupe and ZFS had compression all along.
- ants_everywhere 2y agoIf we're being pedants, then storing the same information in fewer bits than the input is by definition a form of compression, no? (Although yes I understand that file-level compression with a standard algorithm is a different thing than dedup)
- tw04 2y agoThey are not the same thing, but when you boil it down to the raw math, they aren't identical twins, but they're absolutely fraternal twins. Both are trying to eliminate repeating data, it's just the frame of reference that changes. Compression in this context is operating on a given block or handful of blocks. Deduplication is operating on the entire "volume" of data. "Volume" having a different meaning depending on the filesystem/storage array in question.
- UltraSane 2y agoWell put. I like to say compression is just short range dedupe. Hash based dedupe wouldn't be needed if you could just to real-time LZMA on all of the data on a storage array but that just isn't feasible and hash-based dedupe is a very effective compromise.
- ShroudedNight 2y agoIs "paternal twins" a linguistic borrowing of some sort? It seems a relatively novel form of what I've mostly seen referred to as monozygotic / 'identical' twins. Searching for some kind of semi-canonical confirmation of its widespread use turns up one, maybe two articles where it's treated as an orthodox term, and at least an equal number of discussions admonishing its use.
- abrookewood 2y agoCouple of comments. Firstly, you are talking about highly redundant information when referencing VM images (e.g. the C drive on all Windows Serer images will be virtually identical), whereas he was using his own laptop contents as an example. Secondly, I think you are conflating two different features: compression & de-duplication. In ZFS you can have compression turned on (almost always worth it) for a pool, but still have de-duplication disabled.
- UltraSane 2y agoFair point. My experience is with enterprise storage arrays and I have always used dedupe/compression at the same time. Dedupe is going to be a lot less useful on single computers. I consider dedupe/compression to be two different forms of the same thing. compression reduces short range duplication while deduplication reduces long range duplication of data.
- abrookewood 2y agoYeah agreed, very closely related - even more so on ZFS where the compression (AFAIK) is on a block level rather than a file level.
- E39M5S62 2y agoZFS compression is for sure at the block level - it's fully transparent to the userland tools.
- phil21 2y agoBase VM images would be a rare and specific workload. One of the few cases dedupe makes sense. However you are likely using better strategies like block or filesystem cloning if you are doing VM hosting off a ZFS filesystem. Not doing so would be throwing away one of it's primary differentiators as a filesystem in such an environment. General purpose fileserving or personal desktop/laptop use generally has very few duplicated blocks and is not worth the overhead. Backups are hit or miss depending on both how the backups are implemented, and if they are encrypted prior to the filesystem level. Compression is a totally different thing and current ZFS best-practice is to enable it by default for pretty much every workload - the CPU used is barely worth mentioning these days, and the I/O savings can be considerable ignoring any storage space savings. Log storage is going to likely see a lot better than 6:1 savings if you have typical logging, at least in my experience.
- XorNot 2y ago> General purpose fileserving or personal desktop/laptop use generally has very few duplicated blocks and is not worth the overhead. I would contest this is because we don't have a good transparent deduplication right now - just some bad compromises. Hard copies? Edit anything and it gets edited everywhere - not what you want. Symlinks? Look different enough that programs treat them differently. I would argue your regular desktop user actually has an enormous demand for a good deduplicating file system - there's no end of use cases where the first step is "make a separate copy of all the relevant files just in case" and a lot of the time we don't do it because it's just too slow and wasteful of disk space. If you're working with say, large video files, then a good dedupe system would make copies basically instant, and then have a decent enough split algorithm that edit's/cuts/etc. of the type people try to do losslessly or with editing programs are stored efficiently without special effort. How many people are producing video content today? Thanks to Tiktok we've dragged that skill right down to "random teenagers" who might hopefully pipeline into working with larger content.
- armada651 2y agoBut according to the article the regular desktop already has such a dedup system: > If you put all this together, you end up in a place where so long as the client program (like /bin/cp) can issue the right copy offload call, and all the layers in between can translate it (eg the Window application does FSCTL_SRV_COPYCHUNK, which Samba converts to copy_file_range() and ships down to OpenZFS). And again, because there’s that clear and unambiguous signal that the data already exists and also it’s right there, OpenZFS can just bump the refcount in the BRT.
- EasyMark 2y agoI figured he was mostly talking about using dedup on your work (dev machine) computer or family computer at home, not on something like a cloud or streaming server or other back end type operations.
- m463 2y agoI would think VMs qualify as a specific workload, since cloning is almost a given.
- bobmcnamara 2y ago> In my experience 4KB is my preferred block size. This probably has something to do with the VM's filesystem block size. If you have a 4KB filesystem and an 8KB file, the file might be fragmented differently but is still the same 2x4KB blocks just in different places. Now I wonder if filesystems zero the slack space at the end of the last block in a file in hopes of better host compression. Vs leaving it as past bytes.
- Maakuth 2y agoCertainly it makes sense to not have deep copies of VM base images, but the deduplication is not the right way to do it in ZFS. Instead, you can clone the base image and before changes it will take almost no space at all. This is thanks to the copy-on-write nature of ZFS. ZFS deduplication instead tries to find existing copies of data that is being written to the volume. For some use cases it could make a lot of sense (container image storage maybe?), but it's very inefficient if you already know some datasets to be clones of the others, at least initially.
- UltraSane 2y agoWhen a new VM is created from a template on a ZFS file system with dedupe enabled what actually happens? Isn't the ref count of every block of the template simply incremented by one? The only time new data will actually be stored is when a block hash a hash that doesn't already exist.
- Maakuth 2y agoThat's right, though deduplication feature is not the way to do it. The VM template would be a zvol, which is a block device backed by the lower levels of ZFS, and it would be cloned to a new zvol for each VM. Alternatively, if image files were used, the image file could be a reflinked copy. In both cases, new data would be stored only when changes accumulate. Compare this to the deduplication approach: the filesystem would need to keep tabs on the data that's already on disk, identify the case where the same data is being written and then make that a reference to the existing data instead. Very inefficient if on application level you already know that it is just a copy being made. In both of these cases, you could say that the data ends up being deduplicated. But the second approach is what the deduplication feature does. The first one is "just" copy-on-write.
- mrgaro 2y agoFor text based logs I'm almost entirely sure that just using compression is more than enough. ZFS supports compression natively on block level and it's almost always turned on. Trying to use dedup alongside of compression for syslog most likely will not yield any benefits.
- jyounker 2y agoTL;DR; Declares claim that "that feature is only good for specific rare workloads" is odd. Justifies that statement by pointing out that the feature works well of their specific rare workload.