12 ms·
Introduce ZSTD compression to ZFS
- d33 6y agoThere's one thing I don't understand. Each time a new compression algorithm is introduced, it's the Next Big Thing. Why isn't the implementation of the algorithm as simple as linking in the related library, assuming they'd all have a similar interface? After all, it seems like what you need is a header and a function that converts a compressed block to a decompressed one and the other way round. Where's the complexity from the API/implementation perspective given that a library is already there?
- georgyo 6y agoFor one thing, kernel models don't link against libraries. So a new implementation is almost always written.
- Someone 6y agoThis incorporates the zstd source code ‘as is’. FTA: Frequently Asked Questions “Q: Why is it so many lines of code, I can't review all of that... A: Most of this code is the ZSTD library, which has not been altered.” One reason this needs testing because this is a file system. If it breaks, it can lose you much more data than the file being worked on. There also may be serious performance degradation on hardware or configurations the developers didn’t look at. There also is some new code added to call the zstd library.
- tobias3 6y agoIn this case it could use the Linux crypto API, which already has ZSTD support and provides compress/decompress functions. But that is exported as GPLv2 so ZFS needs to do its own thing. And idk if the API is sufficient w.r.t. to e.g. workspace management/reuse. W.r.t. to ZSTD: The usual thing is to provide a zlib compatible API, which ZSTD does ( https://github.com/facebook/zstd/tree/dev/zlibWrapper https://github.com/facebook/zstd/tree/dev/zlibWrapper ). But the ZSTD zlib compatibility layer causes lower performance, so it is better to use it directly. Maybe all future compression algorithms can provide a ZSTD compatible API, so we have to do less work in the future?
- m4rtink 6y agoIt often works that way - when binary RPM compression switched from xz to ZSTD in Fedora recently, it was reaaly just about making sure the tooling can now read ZSTD while keepin xz support in place for compatibility (basically msking sure it links to ZSTD and can use it at runtime). Then just switch the Fedora build system to compress binary RPMs with ZSTD when they are built and you are done. :)
- microcolonel 6y ago> Why isn't the implementation of the algorithm as simple as linking in the related library Because filesystems need to store compression metadata and compressor settings differently than individual archive streams do. Because of the way ZFS stores configuration like this, in the previous version of these patches, they had to choose a subset of the available compression levels when adding zstd support. Different compressors and decompressors also have different state sizes for different settings. Allocating, reusing, and discarding buffers for compression/decompression state in a sensible way inside an operating system kernel is not trivial.
- vkaku 6y agoIt's usually the same as linking, except there are tunables in every library, and the hard part is figuring out which one is appropriate for the job. Maybe, this call is to ask people how this library works out in the real world, on the current datasets that people have. Besides, they are talking of testing implementation specific limitations, and may they want to test for that.
- deleted 6y ago[deleted]
- throw0101a 6y agoSee BSDCan 2018 "Implementing ZSTD in OpenZFS on FreeBSD" by Allan Jude: * https://www.bsdcan.org/2018/schedule/events/947.en.html https://www.bsdcan.org/2018/schedule/events/947.en.html * https://twitter.com/allanjude https://twitter.com/allanjude
- foepys 6y agolz4 is performing pretty well. Better than most zstd-fast settings. It will not be obsolete after adding ZSTD and most likely stay the go-to algorithm if you are unsure what to choose.
- chungy 6y agoI think it's likely that LZ4 remains the default. There are almost no downsides to having LZ4 enabled. zstd is more likely to represent an obsolescence of gzip. It surpasses gzip pretty much always.
- db48x 6y agoJust going by those graphs, I could double my compression ratio by going from lz4 to zstd-1 without going below the speeds the drives in my pool can manage. The usual caveats apply, but it seems to me that this is a pretty good upgrade for the usual case where you're using hard drives instead of fast ssds in a pool.
- snvzz 6y agoThis sounds good for fileservers. But generally speaking, computers usually do more than i/o.
- rini17 6y agoBasically yes, but also depends on if you're I/O bound and have free CPU cycles. If there's limited bandwidth to underlying storage, with zstd it's possible to get more data in/out.
- rakoo 6y agoFYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain, while zstd is for the rest (good (de)compression with slower speed). For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like file transfers for example
- tpetry 6y agoAn option in the future to write data with a fast zfs level so everything is speedy and recompress blocks which have not changed in some time with a more efficient compression ratio would be really great. So you would have almost no performance penalty writing data and very high compression ratio for old data.
- rcthompson 6y agoI think that would require the same architectural changes needed for offline deduplication (i.e. probably not going to happen any time soon, unless I've missed some recent developements).
- deleted 6y ago[deleted]
- cmurf 6y agoFast compression levels of a given algorithm means lower compression ratio. I don't know if ZFS supports variable compression levels (maybe per dataset), but Btrfs ZSTD support uses a mount option, e.g. mount -o compress=zstd:[1-15] Thus it's possible to use a higher level (high compression ratio, slower speed, more CPU and RAM) for e.g. an initial archive. And later use a lower level (or even no compression) when doing updates. Writes use the compression algorithm and level set at mount time; and it's possible to change it while remaining mounted, using -o remount.
- sigstoat 6y ago> I don't know if ZFS supports variable compression levels (maybe per dataset) > Thus it's possible to use a higher level (high compression ratio, slower speed, more CPU and RAM) for e.g. an initial archive. And later use a lower level (or even no compression) when doing updates. yup. works the same in ZFS. you can change the compression setting any time you like, for future writes.
- namibj 6y agoBtrfs is soon getting support for specifying the level on a per-file basis: https://github.com/kdave/btrfs-progs/issues/184 https://github.com/kdave/btrfs-progs/issues/184