10 ms·

It depends on a lot on the compressor. Ancient junk like gzip/deflate only has a 32kb window, more modern compressors like zstd can do 512mb or more. Larger wi

• bscphil 6y ago
  Makes me wonder how tar+zstd would perform for the StackOverflow user. They were using xz, which is hardly ancient junk.
  • loeg 6y ago
    xz is more than ten years old now :-).
    • craftinator 6y ago
      In terms of standard Linux tools, that's hardly ancient.
      • StavrosK 6y ago
        Hell, it's barely gone from "bleeding edge" to mere "cutting edge".
      • Blikkentrekker 6y ago
        Most tools do not become obsolete as quickly. Unlinking a path is hardly something that sees cutting edge improvements, cryptographic hashing, filesystems, and compression is something that is.
  • scaladev 6y ago
    >xz It may not be ancient, but it's certainly not the greatest compression format out there. I found this article to be very informative: https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html (The author worked on lzip so it's probably somewhat biased, but facts are facts.)
    • lifthrasiir 6y ago
      The article calls xz out for its inadequacy as a (long-term) container format, which is nothing to do with its compression method (shared by both xz and lzip). I guess in this thread we are specifically talking about the efficiency of compression algorithms...
      • bscphil 6y ago
        > I guess in this thread we are specifically talking about the efficiency of compression algorithms Yep, and on top of that xz is more efficient at compression than zstd despite being "older", which is the relevant consideration here. I'm not expecting zstd to compete with xz for the StackOverflow case in terms of efficiency, what I want to know is whether file ordering makes the same efficiency difference for zstd as it does for xz.
        • lifthrasiir 6y ago
          > I want to know is whether file ordering makes the same efficiency difference for zstd as it does for xz. It should, because much like other LZ77-based formats zstd uses a distance symbol plus additional uncompressed bits for longer distances, which should be frequent with the unorganized file order. Those formats assume that lower bits of longer distances are essentially random, so if this assumption doesn't hold those bits will affect the efficiency. Zstd also recommends the minimum window size of 8 MB, which is an improvement but also not very large. I've done a quick experiment with a directory hanging around my temporary folder that weighs about 14 MB (Android CellBroadcastReceiver source code, if you ask). The random file order resulted in a file about 1.4% larger than those for the file extension order. So the effect definitely still exists, though I'm not able to quantify that.
    • faho 6y ago
      Most of the points that article makes are utterly irrelevant for most uses of xz, including this one.
• WoodenChair 6y ago
  > Ancient junk like gzip/deflate only has a 32kb window Yes, a reflection of the time that they were first created, but hardly junk. gzip/DEFLATE have saved millions if not billions of hours of human time over the past few decades. Better things exist now, but it's unnecessarily derogatory to call something that significant "junk." And by being industry standards, they have made compression available in many more places than it used to be, even if they are not state of the art. Better to have some reasonable compression than none.
  • dmw_ng 6y ago
    Picture the effect of compression ratios or decompression performance improving 1% globally, or conversely, if planes were replaced with a Wright Brothers model or cars with the T1. When discussing Deflate it's not just 1%, but regularly 200% or 2,000%, nor is it something people use once per month, but practically every time they interact with digital electronics. Regressing to the T1 is unthinkable because they're long out of fashion and widely perceived as obsolete. Every change starts with a shift in perception, a process accelerated by a shift in language. For that I'm content with "junk", it describes exactly how we must feel about this dead technology before we'll ever be rid of it. I'll gladly reminisce once we've escaped its legacy, be it time wasted installing an Android app and the effect it has on the battery (multiplied by billions of users), the time to open an Excel document, power up some smart TVs, fetch an itemized AWS bill over a typical S.E. Asian Internet connection, or grep that bill at speeds appropriate for a state of the art laptop manufactured this side of the millennium. You expect me to pay respect to software that wastes the finite breaths of billions of real people every single day.
    • yongjik 6y ago
      So, I wanted to check so I downloaded linux source tree and ran gzip and zstd. Uncompressed .tar: 1064376320 .xz (downloaded file): 117637692 (88.9%) .gz: 189299444 (82.2%) .gz (level 9): 186533230 (82.5%) .zst: 178012638 (83.3%) .zst --ultra -22: 119676867 (88.8%) So, even at maximum level, zstd is only about 6% better, or about 37% if you consider .gz as baseline, while being about 10x slower than gzip level 9. There are places where having a smaller compressed file is worth the cost of 10x execution time (and much higher memory usage), but serving web pages is almost certainly not one of them. In other words, the reason gzip is still being widely used is simply because it's good enough for its use cases. It's less like the original T1, and more like complaining that people are still driving Toyota Camry 2011 when Toyota Camry 2021 is a so much better car.
• nxpnsv 6y ago
  Exactly because of your caveat they are not junk,