4 ms·

tl;dr: the python lib adds files to the archive sorted by the last modification date by default. this happened to be better in this case, but macOS tar had the

tl;dr: the python lib adds files to the archive sorted by the last modification date by default. this happened to be better in this case, but macOS tar had the same results when using the appropriate sorting flag. Makes me wonder: considering the speed of modern SSDs, would it make sense for compression tools to try various sortings by default, or compare files up front?

tomatocracy 6y ago
Another option is to use something like lrzip as the compressor - not sure if it would have helped in this instance but it incorporates a step which looks for long-range redundancy which can make a huge difference in these sorts of scenarios.
JulianWasTaken 6y ago
Seems better to have some separate higher level metacompression tool which does the heuristic searching across many possible compression formats and parameters.
- duskwuff 6y ago
  This is precisely the approach taken by the Hydra metacompressor in the (closed-source) Oodle compression library: http://cbloomrants.blogspot.com/2017/02/oodle-hydra.html http://cbloomrants.blogspot.com/2017/02/oodle-hydra.html
masklinn 6y ago
> macOS tar had the same results when using the appropriate sorting flag. No, macOS has the same results when using gnutar which provides sorting option. bsdtar (which macOS provides out of the box) has no such option. It just stores files in whichever order you provide them (on the command-line, via the CLI) or it finds them (for recursive tar, so whichever order the directory returns them in).
js2 6y ago
> macOS tar had the same results when using the appropriate sorting flag They had to install and use GNU tar to gain the `--sort` option. macOS (BSD) tar doesn't have it. (You could emulate the behavior by using `find` and passing the member names in on the command-line.)
- mjevans 6y ago
  As mentioned near the end of your comment: typically a workaround (possibly with temporary files instead) E.G. find /path | sort | bsdtar -c -f OUTPUT -n --files-from /dev/stdin ...Args I prefer the second version which terminates paths with the NULL character. Both include -n to not-recurse. find /path -s -print0 | bsdtar -c -f OUTPUT --null -n --files-from /dev/stdin ...Args From the BSD manpage: -s Cause find to traverse the file hierarchies in lexicographical order, i.e., alphabetical order within each directory. Note: `find -s' and `find | sort' may give different results.
formerly_proven 6y ago
Compression is generally CPU bound for most of the typical algorithms (LZMA, zlib/deflate) unless your drive is very very slow (<10 MB/s). There are some quite fast algorithms where this could make sense, basically using something like LZ4 as a proxy for compress-ability. This is the approach some tools use when you tell them to "compress, but only if it makes sense".

tomatocracy 6y ago

Another option is to use something like lrzip as the compressor - not sure if it would have helped in this instance but it incorporates a step which looks for long-range redundancy which can make a huge difference in these sorts of scenarios.

JulianWasTaken 6y ago

Seems better to have some separate higher level metacompression tool which does the heuristic searching across many possible compression formats and parameters.

duskwuff 6y ago
This is precisely the approach taken by the Hydra metacompressor in the (closed-source) Oodle compression library: http://cbloomrants.blogspot.com/2017/02/oodle-hydra.html http://cbloomrants.blogspot.com/2017/02/oodle-hydra.html

masklinn 6y ago

> macOS tar had the same results when using the appropriate sorting flag. No, macOS has the same results when using gnutar which provides sorting option. bsdtar (which macOS provides out of the box) has no such option. It just stores files in whichever order you provide them (on the command-line, via the CLI) or it finds them (for recursive tar, so whichever order the directory returns them in).

js2 6y ago

> macOS tar had the same results when using the appropriate sorting flag They had to install and use GNU tar to gain the `--sort` option. macOS (BSD) tar doesn't have it. (You could emulate the behavior by using `find` and passing the member names in on the command-line.)

mjevans 6y ago
As mentioned near the end of your comment: typically a workaround (possibly with temporary files instead) E.G. find /path | sort | bsdtar -c -f OUTPUT -n --files-from /dev/stdin ...Args I prefer the second version which terminates paths with the NULL character. Both include -n to not-recurse. find /path -s -print0 | bsdtar -c -f OUTPUT --null -n --files-from /dev/stdin ...Args From the BSD manpage: -s Cause find to traverse the file hierarchies in lexicographical order, i.e., alphabetical order within each directory. Note: `find -s' and `find | sort' may give different results.

formerly_proven 6y ago

Compression is generally CPU bound for most of the typical algorithms (LZMA, zlib/deflate) unless your drive is very very slow (<10 MB/s). There are some quite fast algorithms where this could make sense, basically using something like LZ4 as a proxy for compress-ability. This is the approach some tools use when you tell them to "compress, but only if it makes sense".