3 ms·
Which file types are there that benefit from compression, but are not yet compressed as part of the format?
by fh973 9y ago
Which file types are there that benefit from compression, but are not yet compressed as part of the format?
- vetinari 9y agoThe general mix has some potential for compression. Looking at one machine available: * / has 2.08 compression ratio, * /var/cache 2.15 * /var/tmp 1.0 * /home 1.02 * /srv 1.10 (few web apps, pgsql instance for these web apps, svn repo, samba shares, local prometheus store). If your server uses spinning rust, you can also increase the I/O by using compression. You are trading the CPU time for I/O bandwidth. Depending on your workload, it may be a sensible trade-off.
- fh973 9y agoI was thinking more in terms of application data, thank you for giving me this perspective. But still wondering: does the amount of this data really warrant compression? I mean the smallest sensible size of an SSD is around 100GB and it has lots of performance.
- vetinari 9y agoIn this case, the svn repos and samba shares are way over 100GB and even with classic HDDs, the machine can saturate the network and still be basically idling. No need for SSDs, classic drives were more cost effective. The compression was just something nice to have. Lz4 in filesystem is basically for free.
- rleigh 9y agoLogfiles. ZFS with lz4 has given me compression ratios of over 100× with /var/log, due to the huge amount of repetition. My first test of PostgreSQL on ZFS was also quite instructive. lz4 again achieved respectable compression (>10× for my datasets) and improved throughput several fold (with no other tuning!).