8 ms·

How much faster is making a tar archive without gzip?

96 points by edward 4y ago

don-code 4y ago
I'm somewhat surprised, disk being so many orders of magnitudes slower than compute (and to a lesser extent, memory), that there's such a wide gulf. My expectation would have been that the difference would be negligible, as `tar` writes roughly double the bytes (per the article) back to the disk.
jeffbee 4y ago
/usr/bin as input tar czf: 18s (214MB) tar cf: 240ms (553MB) tar c | lz4 -c: 1s (305MB) tar c | zstd -c: 1.6s (214MB) TL;DR gzip is and has been for a long time obsolete as both a format and as a tool. find | cpio: 1.2s Bit disappointed in that last bit, honestly.
- nrclark 4y ago
  Gzip is fine for lots of tasks. It's not the most efficient compressor, either by space or speed. But it has a million implementations that all work well together. If you just need "good enough" compression with no fuss, gzip is still an excellent choice.
  - ericbarrett 4y ago
    I agree with this, even as a long-time zstd user both personally and professionally. The fact is nearly every app, widget, API, and even CPU can encode and decode gzip. Sometimes you need to make the prosaic choice.
  - tiagod 4y ago
    I feel like zstd, if nothing better shows up, will become a lot like gzip in the long run
  - somat 4y ago
    for a long time the status quo was that while there were better compressions engines (bzip2 comes to mind) none were faster than gzip. I actually preferred gzip for a lot of tasks because it was faster to compress and decompress. I saw a comment that zstd is actually faster than gzip so I guess that consideration no longer holds.
    - jeffbee 4y ago
      Most things are faster than gzip, and many of those things have smaller outputs as well. Brotli at level 2 produces the same-sized archive with this input, in just 3.4s. Gzip even at level 1, with a 10% larger archive, still needs 6.5s. zstd, at level 1, produces the same size as gzip -1, in 1 second! zstd, cranked up to 9, makes a 20% smaller archive in 7 seconds, less than half the time of gzip -6.
- ericpauley 4y ago
  Agreed. I’ve switched everything over to zstd and find using it actually speeds most tasks up (lower disk IO).
- koolba 4y ago
  Did you run that more than once to ensure uniform cache hits? Otherwise only the first one will pay the price for the actual disk reads.
- glandium 4y ago
  Recent GNU tar versions have a --zstd flag, FWIW.
  - chungy 4y ago
    It also has an "-a" flag that just runs the right compressor based on output file name. "tar -acf archive.tar.gz" and "tar -acf archive.tar.zst" will do the right thing in both cases.
Not_Oles 4y ago
I can't believe my article is here on HN.
- geocrasher 4y ago
  Congrats! I did some writing on LEB a few years ago and it never made it to HN. So, well done :)
  - Not_Oles 4y ago
    Thanks! Best wishes!
- getcrunk 4y ago
  What is darkstar couldn't find a link
  - spearo77 4y ago
    The way I read the terminal prompts, its the name of his computer.
    - Not_Oles 4y ago
      Exactly!
  - Not_Oles 4y ago
    Darkstar is a traditional name for computers running Slackware. Shameless plug: My gorgeous Darkstar is an antique server on which I sell traditional shell accounts from which one can make "metal" VPSes from the command line. Occasionally somebody who knows 10X or 100X or 1,000X more than I do comes along and becomes a customer. If you are interested, please check metalvps.com. Also, if you like Xv6, please check metalvps.com/Xv6.
    - chasil 4y ago
      The name originally comes from a song by The Greatful Dead. https://m.youtube.com/watch?v=AlWqitKLnfs https://m.youtube.com/watch?v=AlWqitKLnfs These are the lyrics: Dark star crashes, pouring its light into ashes Reason tatters, the forces tear loose from the axis Searchlight casting for faults in the clouds of delusion Shall we go, you and I while we can Through the transitive nightfall of diamonds? - Mirror shatters in formless reflections of matter Glass hand dissolving in ice, petal flowers revolving Lady in velvet recedes in the nights of good-bye Shall we go, you and I while we can Through the transitive nightfall of diamonds?
bluedino 4y ago
Being used to slow broadband links, I always compressed my tar files. You might have wait a half hour like the article, but you'd save an hour on the transfer. And then I used my first gigabit local area network. It was was even more amazing using a 10gb SAN with SSD's. At that point the bottleneck seemed to be all the tiny files on disk (python venv's, for instance)
jedberg 4y ago
The last time I did a test like this, it was way faster to tar first without compression, and then run gzip on the tar file.
- crazytalk 4y ago
  Would be curious about OS / input/output size / hardware/virtualization tech, the only reason I can think for this would be tiny buffers with exorbitantly expensive context switches like you'd see on some older virtualization or e.g. puny escaped-a-VCR ARM chips
- Not_Oles 4y ago
  It's been a few days, but I still had the original files in their original locations. So: root@darkstar:/usr# date -u; time gzip < local-revert.tar > local-revert.tar.gz Tue Oct 11 02:49:32 UTC 2022 real 24m35.214s user 23m57.449s sys 0m27.327s root@darkstar:/usr# From the linked article, for convenient reference, making the .tgz file with tar cvzf took about 28.2 minutes. Making an uncompressed .tar file took about 1.2 minutes. Just now, compressing with gzip took 24.6 minutes. It seems, in this situation, about 2.4 minutes might be saved by creating the tar file first and then compressing separately.
- deleted 4y ago
  [deleted]
usefulcat 4y ago
zstd beats gzip handily, either in terms of speed for a given compression ratio or compression ratio for a given amount of wall clock time. And that's without even using multiple threads, which zstd supports out of the box but for gzip you would need to use a different tool like pigz. And on top of that zstd is also much faster to decompress. In 2022, if you care about compression performance at all, there's no reason to even consider gzip unless you have a specific need for it (e.g. you have to deal with some other thing that can't handle zstd).
- jamal-kumar 4y ago
  Thanks for that information, I wasn't aware until reading into it from your suggestion here that it's what underpins the encryption in btrfs, which is something I've experimented with a bit but didn't realize had zstd as a part of it. Very useful!
- twotwotwo 4y ago
  For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli https://github.com/google/brotli, is another modern format with some good features for high compression levels, which also has Content-Encoding support in web browsers. You might also want to play with compression levels (-1 to -11 or more, zstd's --fast=n); they may not map 1:1 to gzip's. One reason these modern compressors do better is not any particular mistake made defining DEFLATE in the 90s, but that new algos use a few MB of recently seen data as context instead of 32KB, and do other things that would have been impractical in the 90s but are reasonable with modern hardware. The new algorithms also contain lots of smart ideas and have fine-tuned implementations, but that core difference seems important to note.
  - GoblinSlayer 4y ago
    So if you have many connections, you will spend gigabytes of RAM on compression alone? In the age of VM and cloud computing efficiency is important again.
geocrasher 4y ago
I posted this in a comment on the WordPress Hosting FB group in a discussion about whether gzipping a sql file for transport was worth it. I thought it an interesting comparison. This is on an NVMe disk on a 32 core server with a light load. $ time wp db export test.sql real 0m3.410s user 0m2.568s sys 0m0.431s 176M test.sql $ time wp db export - | gzip > file.sql.gz real 0m8.304s user 0m10.010s sys 0m0.439s 40M file.sql.gz $ time wp db export - | gzip --fast > fast.sql.gz real 0m4.196s user 0m5.728s sys 0m0.340s 49M fast.sql.gz The clear winner is gzip --fast. Only 23% slower in speed, but 73% reduction in file size, vs normal's 240% decrease in speed but 88% reduction in file size. 15% smaller file at 240% increased backup time is not worth it.
- cortesoft 4y ago
  > 15% smaller file at 240% increased backup time is not worth it. Doesn't this depend on the bandwidth of the connection the file is being transferred over? You also have to factor in unzipping time.
  - geocrasher 4y ago
    That's true, it does. And on a very fast connection, it's cheaper to leave it uncompressed. And true, I didn't factor in decompression. The context was specifically the speed of backups rather than the speed of restores.
viraptor 4y ago
The whole scheme could use some improvements, tar.gz feels like a bad tool here. 1. If you can, use btrfs/zfs/lvm snapshot and skip the waiting completely. There's no point in duplication. 2. If you do want the copies, use something that can do incremental backup instead. For example Borg can both compress and deduplicate and on top of that will only copy the changes, not the whole directory. 3. (optional) If you're into running custom latest software, you can save some time by using nixpkgs instead, which will also ensure you don't risk collisions in /usr/local. (takes some upfront time investment though)
- tuatoru 4y ago
  > There's no point in duplication. Duplication is precisely the point of backups. Well, multiplication preferably, more than duplication. The old chestnut is that 99% of restores are due to user error. There's still that 1% rump, though.
  - viraptor 4y ago
    In general sure, but not in this context. The author specifically says this is a short term copy done before installation, so it's more of a restore checkpoint than a proper backup. The use case is: > With a handy /usr/local archive, if something goes crazy wrong during my new install, it’s easy to revert.
- smeagull 4y ago
  You know what I like about tar.gz? gzip is often a transparent compression layer on some tools, like say Athena. Tar is a simple concatenation of files + header. Therefore, even though Athena/Hive/Presto doesn't handle tar, I can use tar (or grep) to search the files without having to decompress them.
rwmj 4y ago
One reason to use tar without gzip is that you can random access files within the tarball without unpacking it. tar + xz with --block-size also allows this. https://libguestfs.org/nbdkit-tar-filter.1.html https://libguestfs.org/nbdkit-tar-filter.1.html https://libguestfs.org/nbdkit-xz-filter.1.html https://libguestfs.org/nbdkit-xz-filter.1.html Unfortunately zstd still does not: https://github.com/facebook/zstd/issues/395#issuecomment-535875379 https://github.com/facebook/zstd/issues/395#issuecomment-535...
- st_goliath 4y ago
  > you can random access files within the tarball without unpacking it Kinda sorta. You still need a linear scan first and remember all the offsets, because the tar format does not have an index. > tar + xz with --block-size also allows this. This also requires a linear scan + unpack stage first, with memorizing offsets. Remembering where the XZ blocks start (and corresponding uncompressed offsets) allows a sufficiently smart implementation to pretend that it can seek in the compressed stream, while under the hood seeking to the closest offset it has memorized and uncompressing from there. This is also alluded to in the GitHub issue you linked to. Manually tuning down the block size allows that to be done faster, at the cost of compression. But you still need an unpack/scan phase first. And potentially unpacking and skipping adjacent data when "randomly" accessing a file. Technically, there is nothing stopping an implementation from doing this with gzip/zlib frames as well.
  - rwmj 4y ago
    You only have to scan to the start of the file you want to unpack. In many cases (we use this with Kubernetes CDI) the tarball contains only a few files with the disk image we want to access taking up the majority of the tarball, so in fact a full linear scan is not required and we're only unpacking the first few blocks.
- mxmlnkn 4y ago
  Pragzip actually decompress in parallel and also access at random. I did a Show HN here: https://news.ycombinator.com/item?id=32366959 https://news.ycombinator.com/item?id=32366959 indexed_gzip https://github.com/pauldmccarthy/indexed_gzip https://github.com/pauldmccarthy/indexed_gzip can also do random access but is not parallel. Both have to do a linear scan first though. The implementations however can do the linear scan on-demand, i.e., they scan only as far as needed. bzip2 works very well with this approach. xz only works with this approach when compressed with multiple blocks. Similar is true for zstd. For zstd, there also exists a seekable variant, which stores the block index at the end as metadata to avoid the linear scan. indexed_zstd offers random access to those files https://github.com/martinellimarco/indexed_zstd https://github.com/martinellimarco/indexed_zstd I wrote pragzip and also combined all of the other random access compression backends in ratarmount to offer random access to TAR files that is magnitudes faster than archivemount: https://github.com/mxmlnkn/ratarmount https://github.com/mxmlnkn/ratarmount
rrwo 4y ago
> Second, once we have done compiling a few times, compiling a program from its latest sources can be easier than figuring out how to install an often older version with our distribution’s package manager. This is nonsense.
- rrwo 4y ago
  I'll add that the "older" versions of some utilities in a distribution's package managers often have patches with security fixes from newer versions. The distribution's package manager will identify when there's a newer version, especially with critical security fixes, so I don't have to regularly do manual checks to see if something should be upgraded. Newest isn't always better, and I'm usually happy to be conservative about common system utilities like tar and gzip.
Neil44 4y ago
How about doing the tar first then firing the gzip off as a background job after.
nhoughto 4y ago
Did a bit of testing about how to get files from a remote endpoint unpacked on disk the fastest, most interesting part was developing the method. Some good ideas on this thread I hadn’t thought off to test too. https://nhoughto.github.io/blog/posts/2022/03/09/caching/ https://nhoughto.github.io/blog/posts/2022/03/09/caching/
magicalhippo 4y ago
Sometimes you just want things to go in a hurry. The time to make an archive depends primarily on two factors: processing power and IO speed. Depending on scenario one might have a fast CPU with a slow target device, especially if remote, in which case spending extra time compressing might be a win. In other cases the IO is plenty fast but you're limited by CPU, in which case little or no compression is what you want. Was wondering if any had tried implementing a control loop that would dynamically adjust compression parameters to provide optimal compression speed, utilizing both processing power and IO as best as possible?
djha-skin 4y ago
> Before compiling, I use tar to make a short term backup of how things were before the compile and install. So most of his tar files are mostly the same, having many of the same files in the same order, and 512-block aligned to boot. Due to this block aligning property of tar files, they are well suited to block level desuplication file systems, provided most of their contents are similar to other tar files. I have blogged about using this idea in the context of ci/cd[1], but the idea works just as well with backups. Sharing compression between tar files instead of compressing each tar file might be better anyway. 1: https://blog.djha.skin/blog/cicd-package-compression-could-be-so-much-better/ https://blog.djha.skin/blog/cicd-package-compression-could-b...
simonh 4y ago
Interesting. Back in the 90s I found that compressing files before writing them actually improved performance quite significantly, because disk I/O was so much slower back then. How things change.
mharig 4y ago
You may try sqlite3 -A
jerf 4y ago
"time tar cvzf local-revert.tgz local" This would not negate the point but possibly make the delta even larger, but you have to be careful when benchmarking tar with the "v" flag. The time to print ~400,000 lines out to the terminal can be nontrivial! It may even be the majority of the time for the pure tar case. Like I said, this won't negate the point but amplify it; subtract an approximately constant time from both times and the multiplicative delta between them will grow. (Some modern shells that focus on handling lots of text very quickly may be less impacted than some older-style shells.) In the interests of not making another tiny post, I also endorse the use of parallel compression for the benchmark too, which will close the gap somewhat. You also want to consider the overall use case and the overall bandwidth you have to the final destination. If I were trying to upload something over my home network connection to some remote source, I can afford almost any parallel compression scheme there is because it's a pure win if I can end up sending fewer bytes. Compression is better-than-free in that case. On the flip side, if you don't care about disk usage it's hard to do very much compression at NVMe SSD speeds. (There are some things that may be able to do it, but you won't get much.)
- nijave 4y ago
  Not sure if tar is this way but I think some programs change their buffering behavior based on whether they detect a tty which can further change things
kiririn 4y ago
The article talks about tar.gz for archival, which is scary considering none of tar.gz, tar.xz, tar.zstd are safe formats. Try corrupting a few bytes in the middle and half the archive is lost, even files untouched by the corruption Repair tools only exist for tar.bz2 and tar.lz, but neither gets much use compared to gz/xz. The most safe compression for tar is something that splits the compression into blocks, such as btrfs compression or LTO tape compression
gjvc 4y ago
The real trick is to use a parallelized implementation of the compression tool, viz: tar --use-compress-program /usr/bin/lbzip2 --create --file file.tar.bz2 or if you must use .gz tar --use-compress-program /usr/bin/pigz --create --file file.tar.gz
- sidewndr46 4y ago
  Wow, that's really neat. I have never heard of pigz before despite being an avid Ubuntu user for a long time. Also, I see it supports the "-11" level which is using the zopfli algorithm! Appreciate you sharing.
- digitallyfree 4y ago
  As a FYI pigz also can speed up decompression as well, but the performance boost is minimal. The decompression is still in a single thread but it spawns other threads for i/o and so forth. For compression the speedup is quite linear.
  - optionalsquid 4y ago
    igzip (https://github.com/intel/isa-l https://github.com/intel/isa-l) is much faster than gzip or pigz when it comes to decompression, 2-3x in my experience. There is also a Python module (isal) that provides a GzipFile-like wrapper class, for an easy speed-up of Python scripts that read gzipped files. However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. You also need to make sure to use the latest version if you are going to use it in the context of bioinformatics, since older versions choke on concatenated gzip files common in that field.
    - lifthrasiir 4y ago
      > However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. [EDIT: I misread a benchmark result [1] and wrongly concluded that igzip -3 is better than gzip -9, in reality it is somewhere in the middle of -1 and -9.] [1] https://encode.su/threads/3398-Sequence-Compression-Benchmark?p=66191&viewfull=1#post66191 https://encode.su/threads/3398-Sequence-Compression-Benchmar...

96 points by edward 4y ago

I'm somewhat surprised, disk being so many orders of magnitudes slower than compute (and to a lesser extent, memory), that there's such a wide gulf. My expectation would have been that the difference would be negligible, as `tar` writes roughly double the bytes (per the article) back to the disk.

jeffbee 4y ago

/usr/bin as input tar czf: 18s (214MB) tar cf: 240ms (553MB) tar c | lz4 -c: 1s (305MB) tar c | zstd -c: 1.6s (214MB) TL;DR gzip is and has been for a long time obsolete as both a format and as a tool. find | cpio: 1.2s Bit disappointed in that last bit, honestly.

nrclark 4y ago
Gzip is fine for lots of tasks. It's not the most efficient compressor, either by space or speed. But it has a million implementations that all work well together. If you just need "good enough" compression with no fuss, gzip is still an excellent choice.
- ericbarrett 4y ago
  I agree with this, even as a long-time zstd user both personally and professionally. The fact is nearly every app, widget, API, and even CPU can encode and decode gzip. Sometimes you need to make the prosaic choice.
- tiagod 4y ago
  I feel like zstd, if nothing better shows up, will become a lot like gzip in the long run
- somat 4y ago
  for a long time the status quo was that while there were better compressions engines (bzip2 comes to mind) none were faster than gzip. I actually preferred gzip for a lot of tasks because it was faster to compress and decompress. I saw a comment that zstd is actually faster than gzip so I guess that consideration no longer holds.
  - jeffbee 4y ago
    Most things are faster than gzip, and many of those things have smaller outputs as well. Brotli at level 2 produces the same-sized archive with this input, in just 3.4s. Gzip even at level 1, with a 10% larger archive, still needs 6.5s. zstd, at level 1, produces the same size as gzip -1, in 1 second! zstd, cranked up to 9, makes a 20% smaller archive in 7 seconds, less than half the time of gzip -6.
ericpauley 4y ago
Agreed. I’ve switched everything over to zstd and find using it actually speeds most tasks up (lower disk IO).
koolba 4y ago
Did you run that more than once to ensure uniform cache hits? Otherwise only the first one will pay the price for the actual disk reads.
glandium 4y ago
Recent GNU tar versions have a --zstd flag, FWIW.
- chungy 4y ago
  It also has an "-a" flag that just runs the right compressor based on output file name. "tar -acf archive.tar.gz" and "tar -acf archive.tar.zst" will do the right thing in both cases.

Not_Oles 4y ago

I can't believe my article is here on HN.

geocrasher 4y ago
Congrats! I did some writing on LEB a few years ago and it never made it to HN. So, well done :)
- Not_Oles 4y ago
  Thanks! Best wishes!
getcrunk 4y ago
What is darkstar couldn't find a link
- spearo77 4y ago
  The way I read the terminal prompts, its the name of his computer.
  - Not_Oles 4y ago
    Exactly!
- Not_Oles 4y ago
  Darkstar is a traditional name for computers running Slackware. Shameless plug: My gorgeous Darkstar is an antique server on which I sell traditional shell accounts from which one can make "metal" VPSes from the command line. Occasionally somebody who knows 10X or 100X or 1,000X more than I do comes along and becomes a customer. If you are interested, please check metalvps.com. Also, if you like Xv6, please check metalvps.com/Xv6.
  - chasil 4y ago
    The name originally comes from a song by The Greatful Dead. https://m.youtube.com/watch?v=AlWqitKLnfs https://m.youtube.com/watch?v=AlWqitKLnfs These are the lyrics: Dark star crashes, pouring its light into ashes Reason tatters, the forces tear loose from the axis Searchlight casting for faults in the clouds of delusion Shall we go, you and I while we can Through the transitive nightfall of diamonds? - Mirror shatters in formless reflections of matter Glass hand dissolving in ice, petal flowers revolving Lady in velvet recedes in the nights of good-bye Shall we go, you and I while we can Through the transitive nightfall of diamonds?

bluedino 4y ago

Being used to slow broadband links, I always compressed my tar files. You might have wait a half hour like the article, but you'd save an hour on the transfer. And then I used my first gigabit local area network. It was was even more amazing using a 10gb SAN with SSD's. At that point the bottleneck seemed to be all the tiny files on disk (python venv's, for instance)

jedberg 4y ago

The last time I did a test like this, it was way faster to tar first without compression, and then run gzip on the tar file.

crazytalk 4y ago
Would be curious about OS / input/output size / hardware/virtualization tech, the only reason I can think for this would be tiny buffers with exorbitantly expensive context switches like you'd see on some older virtualization or e.g. puny escaped-a-VCR ARM chips
Not_Oles 4y ago
It's been a few days, but I still had the original files in their original locations. So: root@darkstar:/usr# date -u; time gzip < local-revert.tar > local-revert.tar.gz Tue Oct 11 02:49:32 UTC 2022 real 24m35.214s user 23m57.449s sys 0m27.327s root@darkstar:/usr# From the linked article, for convenient reference, making the .tgz file with tar cvzf took about 28.2 minutes. Making an uncompressed .tar file took about 1.2 minutes. Just now, compressing with gzip took 24.6 minutes. It seems, in this situation, about 2.4 minutes might be saved by creating the tar file first and then compressing separately.
deleted 4y ago
[deleted]

usefulcat 4y ago

zstd beats gzip handily, either in terms of speed for a given compression ratio or compression ratio for a given amount of wall clock time. And that's without even using multiple threads, which zstd supports out of the box but for gzip you would need to use a different tool like pigz. And on top of that zstd is also much faster to decompress. In 2022, if you care about compression performance at all, there's no reason to even consider gzip unless you have a specific need for it (e.g. you have to deal with some other thing that can't handle zstd).

jamal-kumar 4y ago
Thanks for that information, I wasn't aware until reading into it from your suggestion here that it's what underpins the encryption in btrfs, which is something I've experimented with a bit but didn't realize had zstd as a part of it. Very useful!
twotwotwo 4y ago
For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli https://github.com/google/brotli, is another modern format with some good features for high compression levels, which also has Content-Encoding support in web browsers. You might also want to play with compression levels (-1 to -11 or more, zstd's --fast=n); they may not map 1:1 to gzip's. One reason these modern compressors do better is not any particular mistake made defining DEFLATE in the 90s, but that new algos use a few MB of recently seen data as context instead of 32KB, and do other things that would have been impractical in the 90s but are reasonable with modern hardware. The new algorithms also contain lots of smart ideas and have fine-tuned implementations, but that core difference seems important to note.
- GoblinSlayer 4y ago
  So if you have many connections, you will spend gigabytes of RAM on compression alone? In the age of VM and cloud computing efficiency is important again.

geocrasher 4y ago

I posted this in a comment on the WordPress Hosting FB group in a discussion about whether gzipping a sql file for transport was worth it. I thought it an interesting comparison. This is on an NVMe disk on a 32 core server with a light load. $ time wp db export test.sql real 0m3.410s user 0m2.568s sys 0m0.431s 176M test.sql $ time wp db export - | gzip > file.sql.gz real 0m8.304s user 0m10.010s sys 0m0.439s 40M file.sql.gz $ time wp db export - | gzip --fast > fast.sql.gz real 0m4.196s user 0m5.728s sys 0m0.340s 49M fast.sql.gz The clear winner is gzip --fast. Only 23% slower in speed, but 73% reduction in file size, vs normal's 240% decrease in speed but 88% reduction in file size. 15% smaller file at 240% increased backup time is not worth it.

cortesoft 4y ago
> 15% smaller file at 240% increased backup time is not worth it. Doesn't this depend on the bandwidth of the connection the file is being transferred over? You also have to factor in unzipping time.
- geocrasher 4y ago
  That's true, it does. And on a very fast connection, it's cheaper to leave it uncompressed. And true, I didn't factor in decompression. The context was specifically the speed of backups rather than the speed of restores.

viraptor 4y ago

The whole scheme could use some improvements, tar.gz feels like a bad tool here. 1. If you can, use btrfs/zfs/lvm snapshot and skip the waiting completely. There's no point in duplication. 2. If you do want the copies, use something that can do incremental backup instead. For example Borg can both compress and deduplicate and on top of that will only copy the changes, not the whole directory. 3. (optional) If you're into running custom latest software, you can save some time by using nixpkgs instead, which will also ensure you don't risk collisions in /usr/local. (takes some upfront time investment though)

tuatoru 4y ago
> There's no point in duplication. Duplication is precisely the point of backups. Well, multiplication preferably, more than duplication. The old chestnut is that 99% of restores are due to user error. There's still that 1% rump, though.
- viraptor 4y ago
  In general sure, but not in this context. The author specifically says this is a short term copy done before installation, so it's more of a restore checkpoint than a proper backup. The use case is: > With a handy /usr/local archive, if something goes crazy wrong during my new install, it’s easy to revert.
smeagull 4y ago
You know what I like about tar.gz? gzip is often a transparent compression layer on some tools, like say Athena. Tar is a simple concatenation of files + header. Therefore, even though Athena/Hive/Presto doesn't handle tar, I can use tar (or grep) to search the files without having to decompress them.

rwmj 4y ago

One reason to use tar without gzip is that you can random access files within the tarball without unpacking it. tar + xz with --block-size also allows this. https://libguestfs.org/nbdkit-tar-filter.1.html https://libguestfs.org/nbdkit-tar-filter.1.html https://libguestfs.org/nbdkit-xz-filter.1.html https://libguestfs.org/nbdkit-xz-filter.1.html Unfortunately zstd still does not: https://github.com/facebook/zstd/issues/395#issuecomment-535875379 https://github.com/facebook/zstd/issues/395#issuecomment-535...

st_goliath 4y ago
> you can random access files within the tarball without unpacking it Kinda sorta. You still need a linear scan first and remember all the offsets, because the tar format does not have an index. > tar + xz with --block-size also allows this. This also requires a linear scan + unpack stage first, with memorizing offsets. Remembering where the XZ blocks start (and corresponding uncompressed offsets) allows a sufficiently smart implementation to pretend that it can seek in the compressed stream, while under the hood seeking to the closest offset it has memorized and uncompressing from there. This is also alluded to in the GitHub issue you linked to. Manually tuning down the block size allows that to be done faster, at the cost of compression. But you still need an unpack/scan phase first. And potentially unpacking and skipping adjacent data when "randomly" accessing a file. Technically, there is nothing stopping an implementation from doing this with gzip/zlib frames as well.
- rwmj 4y ago
  You only have to scan to the start of the file you want to unpack. In many cases (we use this with Kubernetes CDI) the tarball contains only a few files with the disk image we want to access taking up the majority of the tarball, so in fact a full linear scan is not required and we're only unpacking the first few blocks.
mxmlnkn 4y ago
Pragzip actually decompress in parallel and also access at random. I did a Show HN here: https://news.ycombinator.com/item?id=32366959 https://news.ycombinator.com/item?id=32366959 indexed_gzip https://github.com/pauldmccarthy/indexed_gzip https://github.com/pauldmccarthy/indexed_gzip can also do random access but is not parallel. Both have to do a linear scan first though. The implementations however can do the linear scan on-demand, i.e., they scan only as far as needed. bzip2 works very well with this approach. xz only works with this approach when compressed with multiple blocks. Similar is true for zstd. For zstd, there also exists a seekable variant, which stores the block index at the end as metadata to avoid the linear scan. indexed_zstd offers random access to those files https://github.com/martinellimarco/indexed_zstd https://github.com/martinellimarco/indexed_zstd I wrote pragzip and also combined all of the other random access compression backends in ratarmount to offer random access to TAR files that is magnitudes faster than archivemount: https://github.com/mxmlnkn/ratarmount https://github.com/mxmlnkn/ratarmount

rrwo 4y ago

> Second, once we have done compiling a few times, compiling a program from its latest sources can be easier than figuring out how to install an often older version with our distribution’s package manager. This is nonsense.

rrwo 4y ago
I'll add that the "older" versions of some utilities in a distribution's package managers often have patches with security fixes from newer versions. The distribution's package manager will identify when there's a newer version, especially with critical security fixes, so I don't have to regularly do manual checks to see if something should be upgraded. Newest isn't always better, and I'm usually happy to be conservative about common system utilities like tar and gzip.

Neil44 4y ago

How about doing the tar first then firing the gzip off as a background job after.

nhoughto 4y ago

Did a bit of testing about how to get files from a remote endpoint unpacked on disk the fastest, most interesting part was developing the method. Some good ideas on this thread I hadn’t thought off to test too. https://nhoughto.github.io/blog/posts/2022/03/09/caching/ https://nhoughto.github.io/blog/posts/2022/03/09/caching/

magicalhippo 4y ago

Sometimes you just want things to go in a hurry. The time to make an archive depends primarily on two factors: processing power and IO speed. Depending on scenario one might have a fast CPU with a slow target device, especially if remote, in which case spending extra time compressing might be a win. In other cases the IO is plenty fast but you're limited by CPU, in which case little or no compression is what you want. Was wondering if any had tried implementing a control loop that would dynamically adjust compression parameters to provide optimal compression speed, utilizing both processing power and IO as best as possible?

djha-skin 4y ago

> Before compiling, I use tar to make a short term backup of how things were before the compile and install. So most of his tar files are mostly the same, having many of the same files in the same order, and 512-block aligned to boot. Due to this block aligning property of tar files, they are well suited to block level desuplication file systems, provided most of their contents are similar to other tar files. I have blogged about using this idea in the context of ci/cd[1], but the idea works just as well with backups. Sharing compression between tar files instead of compressing each tar file might be better anyway. 1: https://blog.djha.skin/blog/cicd-package-compression-could-be-so-much-better/ https://blog.djha.skin/blog/cicd-package-compression-could-b...

simonh 4y ago

Interesting. Back in the 90s I found that compressing files before writing them actually improved performance quite significantly, because disk I/O was so much slower back then. How things change.

mharig 4y ago

You may try sqlite3 -A

jerf 4y ago

"time tar cvzf local-revert.tgz local" This would not negate the point but possibly make the delta even larger, but you have to be careful when benchmarking tar with the "v" flag. The time to print ~400,000 lines out to the terminal can be nontrivial! It may even be the majority of the time for the pure tar case. Like I said, this won't negate the point but amplify it; subtract an approximately constant time from both times and the multiplicative delta between them will grow. (Some modern shells that focus on handling lots of text very quickly may be less impacted than some older-style shells.) In the interests of not making another tiny post, I also endorse the use of parallel compression for the benchmark too, which will close the gap somewhat. You also want to consider the overall use case and the overall bandwidth you have to the final destination. If I were trying to upload something over my home network connection to some remote source, I can afford almost any parallel compression scheme there is because it's a pure win if I can end up sending fewer bytes. Compression is better-than-free in that case. On the flip side, if you don't care about disk usage it's hard to do very much compression at NVMe SSD speeds. (There are some things that may be able to do it, but you won't get much.)

nijave 4y ago
Not sure if tar is this way but I think some programs change their buffering behavior based on whether they detect a tty which can further change things

kiririn 4y ago

The article talks about tar.gz for archival, which is scary considering none of tar.gz, tar.xz, tar.zstd are safe formats. Try corrupting a few bytes in the middle and half the archive is lost, even files untouched by the corruption Repair tools only exist for tar.bz2 and tar.lz, but neither gets much use compared to gz/xz. The most safe compression for tar is something that splits the compression into blocks, such as btrfs compression or LTO tape compression

gjvc 4y ago

The real trick is to use a parallelized implementation of the compression tool, viz: tar --use-compress-program /usr/bin/lbzip2 --create --file file.tar.bz2 or if you must use .gz tar --use-compress-program /usr/bin/pigz --create --file file.tar.gz

sidewndr46 4y ago
Wow, that's really neat. I have never heard of pigz before despite being an avid Ubuntu user for a long time. Also, I see it supports the "-11" level which is using the zopfli algorithm! Appreciate you sharing.
digitallyfree 4y ago
As a FYI pigz also can speed up decompression as well, but the performance boost is minimal. The decompression is still in a single thread but it spawns other threads for i/o and so forth. For compression the speedup is quite linear.
- optionalsquid 4y ago
  igzip (https://github.com/intel/isa-l https://github.com/intel/isa-l) is much faster than gzip or pigz when it comes to decompression, 2-3x in my experience. There is also a Python module (isal) that provides a GzipFile-like wrapper class, for an easy speed-up of Python scripts that read gzipped files. However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. You also need to make sure to use the latest version if you are going to use it in the context of bioinformatics, since older versions choke on concatenated gzip files common in that field.
  - lifthrasiir 4y ago
    > However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. [EDIT: I misread a benchmark result [1] and wrongly concluded that igzip -3 is better than gzip -9, in reality it is somewhere in the middle of -1 and -9.] [1] https://encode.su/threads/3398-Sequence-Compression-Benchmark?p=66191&viewfull=1#post66191 https://encode.su/threads/3398-Sequence-Compression-Benchmar...