3 ms·
I used this recently with -0 (no compression) to pack* billions of files into a tar file before sending them over the network. It worked amazing.
by jaimehrubiks 4y ago
I used this recently with -0 (no compression) to pack* billions of files into a tar file before sending them over the network. It worked amazing.
- anderskaseorg 4y agoWhy use tar | pigz -0 when you can just use tar?
- jaimehrubiks 4y agoI used tar --use-compress-program="pigz" to create the tar out of billions of files
- richard_todd 4y agoBut what’s confusing everyone is that tar cf - will create the tar without any external compression program needed.
- koolba 4y agoEven the “f -“ option is unneeded as the default is to stream to stdout. Though it’s always a bit scary to not explicitly specify the destination in case your finger slips and the first target is itself a writeable file.
- jaimehrubiks 4y agoI could definitely be wrong here, apologies for the confusion. I run many of these tasks automated, in some cases I used low compression, in others zero compression. For low compression, that command really shines, for zero compression, I would have bet I also got improvement over regular tar without compression, but again, I could be wrong here. I'll test it again
- ac29 4y agoTar is the archiver here (putting multiple files into one file), pigz with no compression isnt doing anything besides wasting CPU time.
- donatj 4y agoIf you're not going to compress at all, you don't need a compressor at all. All you needed was a .tar and not a .tar.gz
- dividuum 4y agoMaybe I’m missing something, but why send the tar generated stream through a non-compressing compressor when you could just send the tar directly?
- jaimehrubiks 4y agoI didn't have the tar, I created it using: tar --use-compress-program="pigz -0" ...
- gruez 4y agoBut if you don't specify the -z flag when using tar, then it won't be compressed. Why type all that out when omitting one flag does the same thing?
- chlorion 4y agoI think you are a little confused here. Tar is a standalone archive format, you can create tar archives with the tar command and not use any compression utility at all. Here is an example that creates an uncompressed tar archive directly with no compression: tar -cf $directory.tar $directory You can also pipe the created archive to stdout instead of to a file if you want to: tar -c $directory | wc -c # send to a remote system tar -c $directory | ssh dd of=~/$directory.tar You can actually compress the archive by just piping it to a compression command: tar -c $directory | zstd --stdout > $directory.tar.zst The above pipeline is probably very similar to what tar is doing internally when you pass "--use-compression-program". So in your case using "pigz -0" is totally useless since tar creates an uncompressed archive by default. You can totally omit the "--use-compression-command" flag to do what you want.
- jaimehrubiks 4y agoI know about all that. My mistake was thinking that I could take advantage of the multicore nature of pigz even when no compression is being used (I'm not sure if pigz only uses multiple cores for the bzip algorithm, or if it somehow could speed up the packing as well). I'm pretty sure that disk is the bottleneck in most cases, but I wonder if that's not the case always, and a single core could become the bottleneck in very fast IO devices. I want to verify this and test it.