4 ms·

Do you think the transparent compression is worth it? I imagine most of the data people have in their computers is already compressed (images, audios, videos, e

by jolmg 8y ago

Do you think the transparent compression is worth it? I imagine most of the data people have in their computers is already compressed (images, audios, videos, etc.), so it would only slow down reading and writing for most files. I can only see it making sense if much of your disk space is used up by uncompressed text files (or other types of uncompressed filetypes).

shmerl 8y ago
It saves I/O on HDD, which is useful if you have a good multi core CPU. Some things are compressed but not all. BTRFS has some heuristic which switches off compression if it's not saving space. It's not very elaborate now, but they plan to improve it. See https://btrfs.wiki.kernel.org/index.php/Compression#What_happens_to_incompressible_files.3F https://btrfs.wiki.kernel.org/index.php/Compression#What_hap... I'm using zstd compression for my btrfs volumes.
cmurf 8y ago
Using compress=zstd on system root saves me about 35-40%. And I save about 1/2 the space if I use compress-force=zstd mount option. I do this on computers with SDHC/SDXC cards which have dog slow writes and lots of spare CPU time, and some of the time on HDD. I don't use compression on computers with NVMe, it doesn't appear to help much at all, and also I'm pretty sure the compression is currently a single threaded pipe, so for multi-queue devices it's plausible some workloads could become slower. Another use case, soonish I hope to see support for selectable compression levels for zstd (currently only with zlib). And also zstd supports use of a training file for even higher ratios. Both would work well for archiving and seed/sprout use cases, where taking the heavy front end hit with slow writes will be worth it (faster download times, and of course the sprout can have a different compression level or no compression so subsequent writes can be fast. Latest squashfs support includes zstd and also selectable levels, if you need a read-only image.
- shmerl 8y ago
  Do they plan to implement multithreaded compression / decompression?
  - terrelln 8y ago
    Btrfs already supports multithreaded compression and decompression. Each 128 KB block is (de)compressed with a single thread, but multiple blocks can be (de)compressed in parallel. The zstd CLI supports multithreaded compression with the flag `-T <num-threads>`.
    - shmerl 8y ago
      Good to know, thanks! Can you add that info to btrfs wiki compression page, please[1]? 1. https://btrfs.wiki.kernel.org/index.php/Compression https://btrfs.wiki.kernel.org/index.php/Compression
- terrelln 8y ago
  I implemented zstd compression in btrfs/squashfs, and work on upstream zstd. * Btrfs compression is multithreaded, and can use up to the number of cores available on the system. * Compression might not help speed on SSDs, but it should help reduce the burn rate, if you care about that. * My intern wrote a patch to add compression level support to zstd in btrfs [1]. It should be merged upstream soon. * Slightly off topic, but grub will soon understand btrfs compressed with zstd. [1] https://lore.kernel.org/linux-btrfs/20181031181108.289340-1-terrelln@fb.com/ https://lore.kernel.org/linux-btrfs/20181031181108.289340-1-...

by jolmg 8y ago

shmerl 8y ago

It saves I/O on HDD, which is useful if you have a good multi core CPU. Some things are compressed but not all. BTRFS has some heuristic which switches off compression if it's not saving space. It's not very elaborate now, but they plan to improve it. See https://btrfs.wiki.kernel.org/index.php/Compression#What_happens_to_incompressible_files.3F https://btrfs.wiki.kernel.org/index.php/Compression#What_hap... I'm using zstd compression for my btrfs volumes.

cmurf 8y ago

Using compress=zstd on system root saves me about 35-40%. And I save about 1/2 the space if I use compress-force=zstd mount option. I do this on computers with SDHC/SDXC cards which have dog slow writes and lots of spare CPU time, and some of the time on HDD. I don't use compression on computers with NVMe, it doesn't appear to help much at all, and also I'm pretty sure the compression is currently a single threaded pipe, so for multi-queue devices it's plausible some workloads could become slower. Another use case, soonish I hope to see support for selectable compression levels for zstd (currently only with zlib). And also zstd supports use of a training file for even higher ratios. Both would work well for archiving and seed/sprout use cases, where taking the heavy front end hit with slow writes will be worth it (faster download times, and of course the sprout can have a different compression level or no compression so subsequent writes can be fast. Latest squashfs support includes zstd and also selectable levels, if you need a read-only image.

shmerl 8y ago
Do they plan to implement multithreaded compression / decompression?
- terrelln 8y ago
  Btrfs already supports multithreaded compression and decompression. Each 128 KB block is (de)compressed with a single thread, but multiple blocks can be (de)compressed in parallel. The zstd CLI supports multithreaded compression with the flag `-T <num-threads>`.
  - shmerl 8y ago
    Good to know, thanks! Can you add that info to btrfs wiki compression page, please[1]? 1. https://btrfs.wiki.kernel.org/index.php/Compression https://btrfs.wiki.kernel.org/index.php/Compression
terrelln 8y ago
I implemented zstd compression in btrfs/squashfs, and work on upstream zstd. * Btrfs compression is multithreaded, and can use up to the number of cores available on the system. * Compression might not help speed on SSDs, but it should help reduce the burn rate, if you care about that. * My intern wrote a patch to add compression level support to zstd in btrfs [1]. It should be merged upstream soon. * Slightly off topic, but grub will soon understand btrfs compressed with zstd. [1] https://lore.kernel.org/linux-btrfs/20181031181108.289340-1-terrelln@fb.com/ https://lore.kernel.org/linux-btrfs/20181031181108.289340-1-...