2 ms·

I know about all that. My mistake was thinking that I could take advantage of the multicore nature of pigz even when no compression is being used (I'm not sure

I know about all that. My mistake was thinking that I could take advantage of the multicore nature of pigz even when no compression is being used (I'm not sure if pigz only uses multiple cores for the bzip algorithm, or if it somehow could speed up the packing as well). I'm pretty sure that disk is the bottleneck in most cases, but I wonder if that's not the case always, and a single core could become the bottleneck in very fast IO devices. I want to verify this and test it.

chlorion 4y ago
Oh yeah in that case tar itself is still creating the archive with a single thread, the compression gets applied "after" tar creates the archive almost exactly like in the pipeline in my comment. By "after", I don't mean the whole archive is created before the compression part runs, but rather that tar can stream the the archive to stdout as it's being created. Tar needs to be able to do this since it was intended for tape drives, and seeking around on a tape drive is extremely slow. It's actually pretty interesting that we still use tar so widely even though tape drives are not in common use! A new archive format that supports random access during reads (and writes maybe?) would be pretty interesting, but tar works pretty well so there isn't that much of a reason to create alternatives.