44 ms·
Context-aware compression can be a lot of faster than blind filesystem level compression. Some data, like plain text, JSON, or XML, compresses extremely well.
by marginalia_nu 4y ago
Context-aware compression can be a lot of faster than blind filesystem level compression.
Some data, like plain text, JSON, or XML, compresses extremely well. You're almost certainly looking at compression rates of up to 90%. Dense binary formats, especially formats that are already compressed like most image formats won't compress barely at all (sometimes they can even get bigger), and attempting to compress them is an enormous waste of processing power.
- JulianWasTaken 4y agoContinuing the hypothetical without really knowing about today's filesystems, such a system could presumably calculate a moving per page, fd or process- compression factor and turn it down or up if it started seeing lots of incompressible data?
- marginalia_nu 4y agoFor a columnar database, this would probably work fairly well as it would put data of similar compressability adjacent to more of its kind, but more row-based databases where various types of data is intermingled would probably not benefit nearly as much.
- zimpenfish 4y agoTried this on my twitter database's `json` column (ie the JSON response from the API fetching a singular tweet) with a `dict_chooser` of '%Y-%W' to bundle weeks together and it took 985k rows down from 6.6G to 1.5G. Not as good as compressing the whole thing obviously but really handy for a queryable backup.
- cmurf 4y agoAt least in btrfs there's compress vs compress-force mount options. compress uses a cheap estimator of compressibility, whereas compress-force always passes data through the compression algorithm. ZSTD has its own estimator to avoid doing unnecessary work, but I haven't found a comparison of these estimators (which is cheaper). compress results in max extent size of 128K for compressed extents, and 128M for uncompressed data. Whereas compress-force results in 512K max extent size for uncompressed data. While there's a cost to tracking more extents, it might be less than the cost of frequent modification of large extents, particularly if snapshots are used. Of course there's no one answer, it'd be workload and hardware dependant. Hard drives have significant latency impact from non-contiguous extents. NVMe much less impact.
- phiresky 4y agoThe zstd estimator is not that great, so you should definitely keep the auto-detection of the file system on. If you try to compress a completely incompressible file, it will still be extremely slow at higher compression levels: https://github.com/borgbase/vorta/issues/485#issuecomment-885164015 https://github.com/borgbase/vorta/issues/485#issuecomment-88...
- magicalhippo 4y agoI can't recall if they actually implemented or just talked about it, but during the ZFS team meetings there was some discussion about using LZ4 as a "compression canary", to determine if it was worth attempting Zstandard. That is, first try compressing with LZ4, and if it did compress, do actual compression with Zstandard. IIRC the argument went that LZ4 had a very quick bail-out routine, and quite fast compression, so the overall overhead would be quite small, at least for the higher Zstandard compression levels.
- Dagger2 4y agoThey implemented it; it landed in May: https://github.com/openzfs/zfs/commit/f375b23c https://github.com/openzfs/zfs/commit/f375b23c
- telendram 4y agoQuite the reverse actually, the btrfs estimator is way worse than the zstd one. By using the btrfs estimator, you leave a ton of compression ratio on the table, for no good reason.