4 ms·
At least in btrfs there's compress vs compress-force mount options. compress uses a cheap estimator of compressibility, whereas compress-force always passes dat
by cmurf 4y ago
At least in btrfs there's compress vs compress-force mount options. compress uses a cheap estimator of compressibility, whereas compress-force always passes data through the compression algorithm.
ZSTD has its own estimator to avoid doing unnecessary work, but I haven't found a comparison of these estimators (which is cheaper).
compress results in max extent size of 128K for compressed extents, and 128M for uncompressed data. Whereas compress-force results in 512K max extent size for uncompressed data. While there's a cost to tracking more extents, it might be less than the cost of frequent modification of large extents, particularly if snapshots are used. Of course there's no one answer, it'd be workload and hardware dependant. Hard drives have significant latency impact from non-contiguous extents. NVMe much less impact.
- phiresky 4y agoThe zstd estimator is not that great, so you should definitely keep the auto-detection of the file system on. If you try to compress a completely incompressible file, it will still be extremely slow at higher compression levels: https://github.com/borgbase/vorta/issues/485#issuecomment-885164015 https://github.com/borgbase/vorta/issues/485#issuecomment-88...
- magicalhippo 4y agoI can't recall if they actually implemented or just talked about it, but during the ZFS team meetings there was some discussion about using LZ4 as a "compression canary", to determine if it was worth attempting Zstandard. That is, first try compressing with LZ4, and if it did compress, do actual compression with Zstandard. IIRC the argument went that LZ4 had a very quick bail-out routine, and quite fast compression, so the overall overhead would be quite small, at least for the higher Zstandard compression levels.
- Dagger2 4y agoThey implemented it; it landed in May: https://github.com/openzfs/zfs/commit/f375b23c https://github.com/openzfs/zfs/commit/f375b23c
- telendram 4y agoQuite the reverse actually, the btrfs estimator is way worse than the zstd one. By using the btrfs estimator, you leave a ton of compression ratio on the table, for no good reason.