13 ms·

Now using Zstandard instead of xz for package compression

271 points by nloomans 7y ago

ncmncm 7y ago
Zstd has an enormous advantage in compression and, especially, decompression speed. It often doesn't compress quite as much, but we don't care as much as we once did. We rebuild packages more than we once did. This looks like a very good move. Debian should follow suit.
- beatgammit 7y ago
  I build packages periodically from the AUR, and compression is the longest part of the process much of the time. For a while, I disabled compression on AUR packages because it was becoming enough of a problem for me to look into solutions. If it's annoying for me, I can imagine it's especially problematic for package maintainers. I can only imagine how much CPU time switching the compression tool will save.
  - SamWhited 7y ago
    I love the AUR, but every single time I have to wait for it to compress Firefox nightly, and then wait for it to immediately decompress it again because the only reason I was building the package in the first place was to install it I about lose my mind. Hopefully this helps, but I really wish AUR helpers would just disable compression and call it a day so I don't have to go mess with config files that would also change my manual package building routine. EDIT: nevermind, this doesn't seem to have made this the default for building packages locally, just for ones you download from the official repos. Guess I'll go change that by hand and then still be sad that I can't have it easily disabled entirely for AUR helpers but build my packages with compression.
    - beanaroo 7y ago
      This isn't a function of an AUR helper but rather makepkg itself. In makepkg.conf, ommit compression by specifying: PKGEXT='.pkg.tar' More information can be found at https://wiki.archlinux.org/index.php/Makepkg#Tips_and_tricks https://wiki.archlinux.org/index.php/Makepkg#Tips_and_tricks
      - SamWhited 7y ago
        Then it happens for packages I build myself (and want compressed) and those that I just want to install.
        sigotirandolas 7y ago
        You can also override PKGEXT as an environment variable when invoking makepkg, so you can have compression by default but easily skip it when it matters: PKGEXT=.pkg.tar makepkg EDIT: By the way, it's often a big win to use multithreaded compression (pigz, zstd -T0, etc.) in makepkg.conf. With this, it's fast enough that I hardly ever override PKGEXT anymore.
        zerogara 7y ago
        If you care about space more than you care about speed you may want to stick with xz, it is hard to beat or impossible by zstd. So set your own priorities rather than adopt the ones of Arch devs. As long as there will be support within the tools for xz individual builders of packages for their own use can use either or more.
        ncmncm 7y ago
        I doubt anybody today has legitimately enough reason to care about space to prefer xz over zstd. The only plausible reason is that some customer, or customer's customer, is only equipped to unpack archaic formats.
        SamWhited 7y ago
        I don't really care about space all that much, but packages I build tend to get uploaded and the people downloading them may not have fast internet. Meanwhile, packages built by my AUR helper I care about speed (seriously, it takes ages to compress then immediately decompress firefox). The problem isn't that I want to optimize for one or the other, it's that AUR helpers generally have a different need than I do when building my packages myself, but for some reason AUR helpers don't override the compression setting for just their install. Probably due to caching like I said which means they can't assume everyone will want compression off all the time, but I'm not sure, that's just a guess.
    - thenewnewguy 7y ago
      > EDIT: nevermind, this doesn't seem to have made this the default for building packages locally, just for ones you download from the official repos. Guess I'll go change that by hand and then still be sad that I can't have it easily disabled entirely for AUR helpers but build my packages with compression. I believe you can supply it via an environment variable if your AUR helper has the ability to set those for `makepkg`.
      - SamWhited 7y ago
        I'll have to experiment with that, thanks. Still, I don't understand why they don't just do it for me. Maybe they're aggressively caching packages and don't want to take the size hit or something.
  - yjftsjthsd-h 7y ago
    Why did you re-enable it? Seems to work fine in my experience?
G4E 7y ago
For those who want a TLDR : The trade off is 0.8% increase of package size for 1300% increase in decompression speed. Those numbers come from a sample of 542 packages.
- agumonkey 7y ago
  thanks, a great change for those with SSDs
  - crazysim 7y ago
    or CPUs!
    - fctorial 7y ago
      Or RAM
yjftsjthsd-h 7y ago
> If you nevertheless haven't updated libarchive since 2018, all hope is not lost! Binary builds of pacman-static are available from Eli Schwartz' personal repository, signed with their Trusted User keys, with which you can perform the update. I am a little shocked that they bothered; Arch is rolling release and explicitly does not support partial upgrades (https://wiki.archlinux.org/index.php/System_maintenance#Partial_upgrades_are_unsupported https://wiki.archlinux.org/index.php/System_maintenance#Part...). So to hit this means that you didn't update a rather important library for over a year, which officially implies that you didn't update at all for over a year, which... is unlikely to be sensible.
- jpgvm 7y ago
  Arch is actually surprisingly stable and even with infrequent updates on the order of months still upgrades cleanly most of the time. The caveats to this were the great period of instability when switching to systemd, changing the /usr/lib layout, etc but those changes are now pretty far in the past.
  - yjftsjthsd-h 7y ago
    Sure, and I've done partial upgrades and it was mostly fine:) It just surprised me to see the devs going out of their way to support it on volunteer time. On the other hand, maybe that's exactly the reason; maybe someone said "hey look, I can make static packages that are immune to library changes! I guess I'll publish these in case they're useful". Open source is fun like that:)
    - semi-extrinsic 7y ago
      Also, Arch devs probably run Arch servers, and I'd not be surprised if some of those have uptimes in hundreds of days.
      - Foxboron 7y ago
        All Arch infra runs on Arch Linux. The infrastructure repository is all open. https://git.archlinux.org/infrastructure.git/ https://git.archlinux.org/infrastructure.git/ Now, official infra doesn't reach hundreds of days. But personal systems might :)
WinonaRyder 7y ago
Zstandard is awesome! Earlier last year I was doing some research that involved repeatedly grepping through over a terabyte of data, most of which were tiny text files that I had to un-zip/7zip/rar/tar and it was painful (maybe I needed a better laptop). With Zstd I was able to re-compress the whole thing down to a few hundred gigs and use ripgrep which solved the problem beautifully. Out of curiosity I tested compression with (single-threaded) lz4 and found that multi-threaded zstd was pretty close. It was an unscientific and maybe unfair test but I found it amazing that I could get lz4-ish compression speeds at the cost of more CPU but with much better compression ratios. EDIT: Btw, I use arch :) - yes, on servers too.
- bufferoverflow 7y ago
  Here's a compression benchmark. http://pages.di.unipi.it/farruggia/dcb/ http://pages.di.unipi.it/farruggia/dcb/ Looks like Snappy beats both LZ4 and Zstd in compression speed and compression ratio, by a huge margin. LZ4 is a ahead of Snappy in the decompression speed.
  - correct_horse 7y ago
    Similar to how code is read more times than it is written, files are decompressed more times than compressed. I have not researched this opinion much
  - ncmncm 7y ago
    I find these numbers for Snappy entirely implausible. The numbers I know about are wrong: zstd always beats gzip for compression ratio. I will need to do my own testing.
    - ncmncm 7y ago
      I have tested snzip 1.0.4. It compresses about as well as lz4, but more slowly. It also decompresses more slowly. It is faster than zstd -1, but compresses less well. It is possible that it does better with certain kinds of data, but 12x remains implausible. Apparently the current file format has suffix ".sz".
cmurf 7y ago
Fedora 31 switched RPM to use zstd. https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_c... Package installations are quite a bit faster, and while I don't have any numbers I expect that the ISO image compose times are faster, since it performs an installation from RPM to create each of the images. Hopefully in the near future the squashfs image on those ISOs will use zstd, not only for the client side speed boost for boot and install, but it cuts the CPU hit for lzma decompression by a lot (more than 50%). https://pagure.io/releng/issue/8581 https://pagure.io/releng/issue/8581
deleted 7y ago
[deleted]
golergka 7y ago
I used zstd for on-the-fly compression of game data for p2p multiplayer synchronization, and got 2-5x as much data (depends on the payload type) in each TCP packet. Sad that it still doesn't get much adoption in the industry.
- ncmncm 7y ago
  Zstd knows how to use a user-supplied dictionary at each end. I hope you are doing that. But if latency matters you might better use lz4.
  - golergka 7y ago
    Yes, I did. Too bad that I didn't get to see up CI in that project, and current maintainers probably forgot to update the dictionary.
shmerl 7y ago
Was XZ used in parallelized fashion? Otherwise comparing is kind of pointless. Single threaded XZ decompression is way too slow.
- rubicks 7y ago
  I give thanks every day for pxz. I can churn out apt indices so much faster relative to the alternative.
  - shmerl 7y ago
    For general purposes, I like using pixz which is indexable in comparison: https://github.com/vasi/pixz https://github.com/vasi/pixz Do you know if Debian is using parallelized XZ or not with apt / dpkg?
- esotericn 7y ago
  Multithreaded xz is non-deterministic and so it's not a candidate.
  - shmerl 7y ago
    How is it non deterministic? Works pretty consistently for me with pixz.
    - tempay 7y ago
      The bytes of the compressed file are non deterministic and depend on the number of cores used, system load and other “random” factors.
      - shmerl 7y ago
        Can't you set those parameters during compression to something fixed? Should be doable.
        7y ago
filereaper 7y ago
Apparently this is how to use Zstd with tar if anyone else was wondering: tar -I zstd -xvf archive.tar.zst https://stackoverflow.com/questions/45355277/how-can-i-decompress-an-archive-file-having-tar-zst https://stackoverflow.com/questions/45355277/how-can-i-decom... Hopefully there's another option added to tar that simplifies this if this compression becomes mainstream.
- viraptor 7y ago
  tar accepts `-a` for format autodetection for a while now. You can do: tar -axf archive.tar.whatever and it should work for gz, bz2, Z, zstd, and probably more. (verified works for zstd on gnu tar 1.32)
  - yjftsjthsd-h 7y ago
    I think that's a GNU extension, so obviously fine on Arch, but probably not on ex. MacOS (Darwin) or Alpine (busybox) by default.
    - JonathonW 7y ago
      BSD tar on my Mac (running 10.15.2) has -a for tarfile creation (-c mode); it always autodetects the compression format in extraction mode (-z, -j, etc. are ignored if -x is specified). Not sure when either behavior would've been introduced; the somewhat-older machine I tested on (running 10.13) does not have -a but does have the autodetection behavior on extract. -I, on the other hand (which, in gnutar, specifies a compression program to run the output through), appears to actually be GNU-specific. BSD tar makes -I synonymous with -T (specifying a file containing the list of filenames to be extracted or archived). (Please don't use Zstandard if you care about cross-platform compatibility at all, though-- it's fine in controlled environments like an OS package manager, but I don't have it on my Mac, nor do I have it by default on my Ubuntu server (which is still sitting back on 16.04; I should fix that sooner or later).)
      - Lammy 7y ago
nwah1 7y ago
I wonder if they will switch to using zstd for mkinititcpio
- yjftsjthsd-h 7y ago
  I thought that was user configurable? Or do you mean by default?
  - nwah1 7y ago
    Yea, by default. Last time I tried it manually, the kernel wouldn't boot. Best to have these things handled for you.
- Squithrilve 7y ago
  mkinitcpio is being replaced with dracut so zstd won't probably happen.
  - nwah1 7y ago
    Man page says zstd is an option on dracut http://man7.org/linux/man-pages/man5/dracut.conf.5.html http://man7.org/linux/man-pages/man5/dracut.conf.5.html
    - Foxboron 7y ago
      The kernel doesn't support booting zstd compressed initramfs' yet, but you can very well use zstd compression with dracut and mkinitcpio
  - Foxboron 7y ago
    Well, that is a bit on a 50/50 coinflip currently. There has been intentions but we need some collaboration from upstream to make this happen. It's not set in stone currently.
dhsysusbsjsi 7y ago
Quick shout out to LZFSE. Similar compression ratio to zlib but much faster. https://github.com/lzfse/lzfse https://github.com/lzfse/lzfse
Lammy 7y ago
Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours ago?
- Dylan16807 7y ago
  Most of the text on the front page is that size and that color of gray. If it's not easy to read, then the problem is between the css and your screen. Not the title rules.
  - Lammy 7y ago
    I'm talking about the difference in size and contrast between the actual headline and the trailing HN sitebit "(archlinux.org)". The final size they end up on my screen is irrelevant to my point, because my point is about the size and contrast difference _between_ the two, whatever final sizes those might happen to be. The default HN stylesheet calls for 10pt and 8pt for those, respectively, so it's not like I'm just making this up. I'm saying the verbatim title rule is a poor fit here because it took a relevant (central, even!) part of the headline and moved it to a spot of secondary importance and size. There are cases where I defend the rule, but right now I am talking about this case and only this case :)
    - Dylan16807 7y ago
      So even if the url got bumped to 12pt, you'd complain if the rest was 15? I think that's weird. As long as it's on the same line as the title and easily legible, I really don't see a problem. And it's not a spot of secondary importance. If it was still in the title, making it longer, the spot where you see the url would have title in it.
      - pbhjpbhj 7y ago
maxpert 7y ago
I’ve used LZ4 and Snappy in production for compressing cache/mq payloads. This is on a service serving billions of clicks in a day. So far very happy with the results, I know zstd requires more CPU than LZ4 or snappy on average but has someone used it under heavy traffic loads on web services. I am really interested trying it out but at the same time held back by “don’t fix it if it ain’t broken”.
- loeg 7y ago
  Zstd has "fast" negative levels (-5, -4, ... -1, 1, ..., 22). -4 or -5 are purportedly comparable (but not quite as good) as LZ4.
  - ncmncm 7y ago
    Better to just use Lz4, then.
    - emn13 7y ago
      Maybe. The thing is; zstd is quite close, and unlike lz4, zstd has a broad curve of supported speed/time tradeoffs. Unless you're huge and engineering effort is essentially free or at least the microoptimization for one specific ratio is worth the tradeoff - you may be better off choosing the solution that's less opinionated about the settings. If it then turns out that you care mostly about decompression speed + compression ratio and a little less about compression speed, it's trivial to go there. Or maybe it turns out you only sometimes need the speed, but usually can afford spending a little more CPU time - so you default to higher compression ratios, but under load use lower ones (there's even a streaming mode built-in that does this for you for large streams). Or maybe your dataset is friendly to the parallization options, and zstd actually outperforms lz4. If you know your use case well and are sure the situation won't change (or don't mind swapping compression algorithms when they do), then lz4 still has a solid niche, especially where compression speed matters more than decompression speed. But in many if not most cases I'd say it's probably a kind of premature optimization at this point, even if you think you're close to lz4's sweet spot.
- 7y ago
kbumsik 7y ago
> Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup. Impressive. As a AUR package maintainer I am also wondering how the compression speed is though.
- ncmncm 7y ago
  Compression speed is many, many, many times faster than xz, and (only) much faster than gzip. Really, only lz4 beats it.
  - integricho 7y ago
    After reading these comments, I can't help but wonder, what is the benefit of Zstd over lz4? Why didn't they switch to lz4 if it was the speed of the algorithm that they favored even with marginally worse compression ratios?
    - isatty 7y ago
      Guessing that 0.8x size increase for 1300% speedup was worth the tradeoff but maybe ≥1.5 size increase or more was not (especially considering a 1300%->2000% increase is not going to be user visible for 99% of the packages).
      - TJSomething 7y ago
        It's not 0.8 times size increase, it's a 0.008 times size increase, since the unit is percent. The latter seems pretty marginal to me.
    - ncmncm 7y ago
      Where Zstd will reduce, say, 3x, Lz4 reduces only 2.5x. This doesn't seem very different until you look at it from the other end: my .zst file is 3.3 GB, but the .lz4 would have been 4 GB, which is 700 MB more. Was a time when 700 MB mattered; it was as much as you could get onto a CD. So, there is a place for each. I would set up the process to use Lz4 when testing, and Zstd for actual delivery to download archives. In some circumstances, particularly when using a shared file server, Lz4 can be quite a lot faster than writing and reading data uncompressed.
loeg 7y ago
I'd love to see Zstandard accepted in other places where the current option is only the venerable zlib. E.g., git packing, ssh -C. It's got more breadth and is better (ratio / cpu) than zlib at every point in the curve where zlib even participates.
- ncmncm 7y ago
  Wireshark! Wireshark! Also lz4, of course.
- jacobolus 7y ago
  It would be great to see better compression supported by browsers.
  - tyingq 7y ago
    Chrome apparently tested zstd out, and it's an improvement over Chrome's forked and optimized zlib on x64, but slower on ARM/Android. https://bugs.chromium.org/p/chromium/issues/detail?id=912902#c6 https://bugs.chromium.org/p/chromium/issues/detail?id=912902... Few (or none?) of Chrome's fairly dramatic improvements to zlib have been upstreamed. https://github.com/madler/zlib/issues/346 https://github.com/madler/zlib/issues/346 Edit: Also, if browsers do adopt zstd, it's likely you'll end up with the same situation where they fork their own implementation of zstd. Upstreaming requires signing Facebook's CLA, which has patent clauses that don't work for most.
  - dictum 7y ago
    Brotli has wide browser support (https://caniuse.com/#feat=brotli https://caniuse.com/#feat=brotli) and comes closer to zstd in compression ratio and compression speed, but its decompression speed is significantly lower and closer to zlib. https://github.com/facebook/zstd#benchmarks https://github.com/facebook/zstd#benchmarks AFAIK (I haven't looked much into it since 2018) it's not widely supported by CDNs, but at least Cloudflare seems to serve it by default (EDIT: must be enabled per-site https://support.cloudflare.com/hc/en-us/articles/200168396-What-will-Cloudflare-compress- https://support.cloudflare.com/hc/en-us/articles/200168396-W...)
Annatar 7y ago
I couldn't care less about decompression speed, because the bottleneck is the network, which means that I want my packages as small as possible. Smaller packages mean faster installation; at 54 MB/s or faster decompression rate of xz, I couldn't care less about a few milliseconds saved during decompression. For me, this decision is dumbass stupid.
- ubercow13 7y ago
  Why do you care so much about the few extra miliseconds wasted downloading, then? (0.8% size increase is ~ 0). Also don't forget that Arch can also be used on machines with very slow CPU but very fast network connections, like many VPSs. I think this will make a tangible difference on mine. This is also a big improvement for package maintainers and anyone building their own packages without bothering to modify the makepkg defaults, eg. most people using an AUR helper.
  - Annatar 7y ago
    Because size does matter.
- snvzz 7y ago
  The extra decompression complexity might be a joke on a Zen2 server, but it definitely is not in older systems. If this was netbsd m68k, you'd probably easily understand.
  - Annatar 7y ago
    I use xz on my A1200 all of the time, and Amiga is the stereotypical system where maximum possible compression matters over everything else. Don't make assumptions about me.
    - snvzz 7y ago
      I applaud your patience. Even with my vampire, I'll use something faster whether at all possible. May I ask, why xz over, say, PAQ8PF?
Phlogi 7y ago
The wiki is already up to date if you build your own or AUR packages and want to use multiple cpu cores https://wiki.archlinux.org/index.php/Makepkg#Utilizing_multiple_cores_on_compression https://wiki.archlinux.org/index.php/Makepkg#Utilizing_multi...
rwmj 7y ago
I wish zstd supported seeking and partial decompression (https://github.com/facebook/zstd/issues/395#issuecomment-535875379 https://github.com/facebook/zstd/issues/395#issuecomment-535...). We could then use it for hosting disk images as it would be a lot faster than xz which we currently use.
- ncmncm 7y ago
  Fun fact: Two zstd files appended is a zstd file. Also, parallel zstd must have some way to split up the work, that you could maybe use too.
  - rwmj 7y ago
    I would suggest reading the github issue that I linked to, you'll see why it's not currently possible.
vmchale 7y ago
What of lzip?
gravitas 7y ago
AUR users -- the default settings in /etc/makepkpg.conf (delivered by the pacman package as of 5.2.1-1) are still at xz, you must manually edit your local config: PKGEXT='.pkg.tar.zst' The largest package I always wait on perfect for this scenario is `google-cloud-sdk` (the re-compression is a killer -- `zoom` is another one in AUR that's a beast) so I used it as a test on my laptop here in "real world conditions" (browsers running, music playing, etc.). It's an old Dell m4600 (i7-2760QM, rotating disk), nothing special. What matters is using default xz, compression takes twice as long and appears to drive the CPU harder. Using xz my fans always kick in for a bit (normal behaviour), testing zst here did not kick the fans on the same way. After warming up all my caches with a few pre-builds to try and keep it fair by reducing disk I/O, here's a sampling of the results: xz defaults - Size: 33649964 real 2m23.016s user 1m49.340s sys 0m35.132s ---- zst defaults - Size: 47521947 real 1m5.904s user 0m30.971s sys 0m34.021s ---- zst mpthread - Size: 47521114 real 1m3.943s user 0m30.905s sys 0m33.355s I can re-run them and get a pretty consistent return (so that's good, we're "fair" to a degree); there's disk activity building this package (seds, etc.) so it's not pure compression only. It's a scenario I live every time this AUR package (google-cloud-sdk) is refreshed and we get to upgrade. Trying to stick with real world, not synthetic benchmarks. :) I did not seem to notice any appreciable difference in adding the `--threads=0` to `COMPRESSZST=` (from the Arch wiki), they both consistently gave me right around what you see above. This was compression only testing which is where my wait time is when upgrading these packages, huge improvement with zst seen here...
- Foxboron 7y ago
  It should be noted that the makepkg.conf file distributed with pacman does not contain the same compression settings as the one used to build official packages. pacman: COMPRESSZST=(zstd -c -z -q -) https://git.archlinux.org/svntogit/packages.git/tree/trunk/makepkg.conf?h=packages/pacman#n133 https://git.archlinux.org/svntogit/packages.git/tree/trunk/m... devtools: COMPRESSZST=(zstd -c -T0 --ultra -20 -) https://github.com/archlinux/devtools/blob/master/makepkg-x86_64.conf#L135 https://github.com/archlinux/devtools/blob/master/makepkg-x8...
  - gravitas 7y ago
m4rtink 7y ago
BTW, Fedora recently switched to zstd compression for its packages as well. For the same resons basically - much better overall de/compression speed while keeping the result mostly the same size. Also one more benefit of zstd compression, that is not widely noted - a zstd file conpressed with multiple threads is binary the same as file compressed with single thread. So you can use multi threaded compression and you will end up with the same file cheksum, which is very important for package signing. On the other hand xz, which has been used before, produces a binary different file if compressed by single or multiple threads. This basucally precludes multi threaded compression at package build time, as the compressed file checksums would not match if the package was rebuild with a different number of compression threads. (the unpacked payload will be always the same, but the compressed xz file will be binary different)
imtringued 7y ago
This blog post probably wasted more of my time than I will ever gain from the faster decompression...
JeremyNT 7y ago
I learned about this one the hard way when I went to update a really crufty (~ 1 year since last update) Arch system I use infrequently the other day. I had failed to update my libarchive version prior to the change and the package manager could not process the new format. Luckily updating libarchive manually with an intermediate version resolved my issue and everything proceeded fine. This is a good change, but it's a reminder to pay attention to the Arch Linux news feed, because every now and then something important will change. The maintainers provided ample warning about this change there (and indeed I had updated by other systems in response) so we procrastinators really had no excuse :)
zerogara 7y ago
Most of the results published show very little positive or negative speed in decompression, where is all this -1300% coming from? edit: Sorry, my fault that was decompression RAM I was thinking about, not speed, although I was influenced by my test that without measuring both xz and zstd seemed instant.

271 points by nloomans 7y ago

Zstd has an enormous advantage in compression and, especially, decompression speed. It often doesn't compress quite as much, but we don't care as much as we once did. We rebuild packages more than we once did. This looks like a very good move. Debian should follow suit.

beatgammit 7y ago
I build packages periodically from the AUR, and compression is the longest part of the process much of the time. For a while, I disabled compression on AUR packages because it was becoming enough of a problem for me to look into solutions. If it's annoying for me, I can imagine it's especially problematic for package maintainers. I can only imagine how much CPU time switching the compression tool will save.
- SamWhited 7y ago
  I love the AUR, but every single time I have to wait for it to compress Firefox nightly, and then wait for it to immediately decompress it again because the only reason I was building the package in the first place was to install it I about lose my mind. Hopefully this helps, but I really wish AUR helpers would just disable compression and call it a day so I don't have to go mess with config files that would also change my manual package building routine. EDIT: nevermind, this doesn't seem to have made this the default for building packages locally, just for ones you download from the official repos. Guess I'll go change that by hand and then still be sad that I can't have it easily disabled entirely for AUR helpers but build my packages with compression.
  - beanaroo 7y ago
    This isn't a function of an AUR helper but rather makepkg itself. In makepkg.conf, ommit compression by specifying: PKGEXT='.pkg.tar' More information can be found at https://wiki.archlinux.org/index.php/Makepkg#Tips_and_tricks https://wiki.archlinux.org/index.php/Makepkg#Tips_and_tricks
    - SamWhited 7y ago
      Then it happens for packages I build myself (and want compressed) and those that I just want to install.
      - sigotirandolas 7y ago
        You can also override PKGEXT as an environment variable when invoking makepkg, so you can have compression by default but easily skip it when it matters: PKGEXT=.pkg.tar makepkg EDIT: By the way, it's often a big win to use multithreaded compression (pigz, zstd -T0, etc.) in makepkg.conf. With this, it's fast enough that I hardly ever override PKGEXT anymore.
      - zerogara 7y ago
        If you care about space more than you care about speed you may want to stick with xz, it is hard to beat or impossible by zstd. So set your own priorities rather than adopt the ones of Arch devs. As long as there will be support within the tools for xz individual builders of packages for their own use can use either or more.
        ncmncm 7y ago
        I doubt anybody today has legitimately enough reason to care about space to prefer xz over zstd. The only plausible reason is that some customer, or customer's customer, is only equipped to unpack archaic formats.
        SamWhited 7y ago
        I don't really care about space all that much, but packages I build tend to get uploaded and the people downloading them may not have fast internet. Meanwhile, packages built by my AUR helper I care about speed (seriously, it takes ages to compress then immediately decompress firefox). The problem isn't that I want to optimize for one or the other, it's that AUR helpers generally have a different need than I do when building my packages myself, but for some reason AUR helpers don't override the compression setting for just their install. Probably due to caching like I said which means they can't assume everyone will want compression off all the time, but I'm not sure, that's just a guess.
  - thenewnewguy 7y ago
    > EDIT: nevermind, this doesn't seem to have made this the default for building packages locally, just for ones you download from the official repos. Guess I'll go change that by hand and then still be sad that I can't have it easily disabled entirely for AUR helpers but build my packages with compression. I believe you can supply it via an environment variable if your AUR helper has the ability to set those for `makepkg`.
    - SamWhited 7y ago
      I'll have to experiment with that, thanks. Still, I don't understand why they don't just do it for me. Maybe they're aggressively caching packages and don't want to take the size hit or something.
- yjftsjthsd-h 7y ago
  Why did you re-enable it? Seems to work fine in my experience?

G4E 7y ago

For those who want a TLDR : The trade off is 0.8% increase of package size for 1300% increase in decompression speed. Those numbers come from a sample of 542 packages.

agumonkey 7y ago
thanks, a great change for those with SSDs
- crazysim 7y ago
  or CPUs!
  - fctorial 7y ago
    Or RAM

yjftsjthsd-h 7y ago

> If you nevertheless haven't updated libarchive since 2018, all hope is not lost! Binary builds of pacman-static are available from Eli Schwartz' personal repository, signed with their Trusted User keys, with which you can perform the update. I am a little shocked that they bothered; Arch is rolling release and explicitly does not support partial upgrades (https://wiki.archlinux.org/index.php/System_maintenance#Partial_upgrades_are_unsupported https://wiki.archlinux.org/index.php/System_maintenance#Part...). So to hit this means that you didn't update a rather important library for over a year, which officially implies that you didn't update at all for over a year, which... is unlikely to be sensible.

jpgvm 7y ago
Arch is actually surprisingly stable and even with infrequent updates on the order of months still upgrades cleanly most of the time. The caveats to this were the great period of instability when switching to systemd, changing the /usr/lib layout, etc but those changes are now pretty far in the past.
- yjftsjthsd-h 7y ago
  Sure, and I've done partial upgrades and it was mostly fine:) It just surprised me to see the devs going out of their way to support it on volunteer time. On the other hand, maybe that's exactly the reason; maybe someone said "hey look, I can make static packages that are immune to library changes! I guess I'll publish these in case they're useful". Open source is fun like that:)
  - semi-extrinsic 7y ago
    Also, Arch devs probably run Arch servers, and I'd not be surprised if some of those have uptimes in hundreds of days.
    - Foxboron 7y ago
      All Arch infra runs on Arch Linux. The infrastructure repository is all open. https://git.archlinux.org/infrastructure.git/ https://git.archlinux.org/infrastructure.git/ Now, official infra doesn't reach hundreds of days. But personal systems might :)

WinonaRyder 7y ago

Zstandard is awesome! Earlier last year I was doing some research that involved repeatedly grepping through over a terabyte of data, most of which were tiny text files that I had to un-zip/7zip/rar/tar and it was painful (maybe I needed a better laptop). With Zstd I was able to re-compress the whole thing down to a few hundred gigs and use ripgrep which solved the problem beautifully. Out of curiosity I tested compression with (single-threaded) lz4 and found that multi-threaded zstd was pretty close. It was an unscientific and maybe unfair test but I found it amazing that I could get lz4-ish compression speeds at the cost of more CPU but with much better compression ratios. EDIT: Btw, I use arch :) - yes, on servers too.

bufferoverflow 7y ago
Here's a compression benchmark. http://pages.di.unipi.it/farruggia/dcb/ http://pages.di.unipi.it/farruggia/dcb/ Looks like Snappy beats both LZ4 and Zstd in compression speed and compression ratio, by a huge margin. LZ4 is a ahead of Snappy in the decompression speed.
- correct_horse 7y ago
  Similar to how code is read more times than it is written, files are decompressed more times than compressed. I have not researched this opinion much
- ncmncm 7y ago
  I find these numbers for Snappy entirely implausible. The numbers I know about are wrong: zstd always beats gzip for compression ratio. I will need to do my own testing.
  - ncmncm 7y ago
    I have tested snzip 1.0.4. It compresses about as well as lz4, but more slowly. It also decompresses more slowly. It is faster than zstd -1, but compresses less well. It is possible that it does better with certain kinds of data, but 12x remains implausible. Apparently the current file format has suffix ".sz".

cmurf 7y ago

Fedora 31 switched RPM to use zstd. https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_c... Package installations are quite a bit faster, and while I don't have any numbers I expect that the ISO image compose times are faster, since it performs an installation from RPM to create each of the images. Hopefully in the near future the squashfs image on those ISOs will use zstd, not only for the client side speed boost for boot and install, but it cuts the CPU hit for lzma decompression by a lot (more than 50%). https://pagure.io/releng/issue/8581 https://pagure.io/releng/issue/8581

deleted 7y ago

[deleted]

golergka 7y ago

I used zstd for on-the-fly compression of game data for p2p multiplayer synchronization, and got 2-5x as much data (depends on the payload type) in each TCP packet. Sad that it still doesn't get much adoption in the industry.

ncmncm 7y ago
Zstd knows how to use a user-supplied dictionary at each end. I hope you are doing that. But if latency matters you might better use lz4.
- golergka 7y ago
  Yes, I did. Too bad that I didn't get to see up CI in that project, and current maintainers probably forgot to update the dictionary.

shmerl 7y ago

Was XZ used in parallelized fashion? Otherwise comparing is kind of pointless. Single threaded XZ decompression is way too slow.

rubicks 7y ago
I give thanks every day for pxz. I can churn out apt indices so much faster relative to the alternative.
- shmerl 7y ago
  For general purposes, I like using pixz which is indexable in comparison: https://github.com/vasi/pixz https://github.com/vasi/pixz Do you know if Debian is using parallelized XZ or not with apt / dpkg?
esotericn 7y ago
Multithreaded xz is non-deterministic and so it's not a candidate.
- shmerl 7y ago
  How is it non deterministic? Works pretty consistently for me with pixz.
  - tempay 7y ago
    The bytes of the compressed file are non deterministic and depend on the number of cores used, system load and other “random” factors.
    - shmerl 7y ago
      Can't you set those parameters during compression to something fixed? Should be doable.
      - 7y ago

filereaper 7y ago

Apparently this is how to use Zstd with tar if anyone else was wondering: tar -I zstd -xvf archive.tar.zst https://stackoverflow.com/questions/45355277/how-can-i-decompress-an-archive-file-having-tar-zst https://stackoverflow.com/questions/45355277/how-can-i-decom... Hopefully there's another option added to tar that simplifies this if this compression becomes mainstream.

viraptor 7y ago
tar accepts `-a` for format autodetection for a while now. You can do: tar -axf archive.tar.whatever and it should work for gz, bz2, Z, zstd, and probably more. (verified works for zstd on gnu tar 1.32)
- yjftsjthsd-h 7y ago
  I think that's a GNU extension, so obviously fine on Arch, but probably not on ex. MacOS (Darwin) or Alpine (busybox) by default.
  - JonathonW 7y ago
    BSD tar on my Mac (running 10.15.2) has -a for tarfile creation (-c mode); it always autodetects the compression format in extraction mode (-z, -j, etc. are ignored if -x is specified). Not sure when either behavior would've been introduced; the somewhat-older machine I tested on (running 10.13) does not have -a but does have the autodetection behavior on extract. -I, on the other hand (which, in gnutar, specifies a compression program to run the output through), appears to actually be GNU-specific. BSD tar makes -I synonymous with -T (specifying a file containing the list of filenames to be extracted or archived). (Please don't use Zstandard if you care about cross-platform compatibility at all, though-- it's fine in controlled environments like an OS package manager, but I don't have it on my Mac, nor do I have it by default on my Ubuntu server (which is still sitting back on 16.04; I should fix that sooner or later).)
    - Lammy 7y ago

nwah1 7y ago

I wonder if they will switch to using zstd for mkinititcpio

yjftsjthsd-h 7y ago
I thought that was user configurable? Or do you mean by default?
- nwah1 7y ago
  Yea, by default. Last time I tried it manually, the kernel wouldn't boot. Best to have these things handled for you.
Squithrilve 7y ago
mkinitcpio is being replaced with dracut so zstd won't probably happen.
- nwah1 7y ago
  Man page says zstd is an option on dracut http://man7.org/linux/man-pages/man5/dracut.conf.5.html http://man7.org/linux/man-pages/man5/dracut.conf.5.html
  - Foxboron 7y ago
    The kernel doesn't support booting zstd compressed initramfs' yet, but you can very well use zstd compression with dracut and mkinitcpio
- Foxboron 7y ago
  Well, that is a bit on a 50/50 coinflip currently. There has been intentions but we need some collaboration from upstream to make this happen. It's not set in stone currently.

dhsysusbsjsi 7y ago

Quick shout out to LZFSE. Similar compression ratio to zlib but much faster. https://github.com/lzfse/lzfse https://github.com/lzfse/lzfse

Lammy 7y ago

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours ago?

Dylan16807 7y ago
Most of the text on the front page is that size and that color of gray. If it's not easy to read, then the problem is between the css and your screen. Not the title rules.
- Lammy 7y ago
  I'm talking about the difference in size and contrast between the actual headline and the trailing HN sitebit "(archlinux.org)". The final size they end up on my screen is irrelevant to my point, because my point is about the size and contrast difference _between_ the two, whatever final sizes those might happen to be. The default HN stylesheet calls for 10pt and 8pt for those, respectively, so it's not like I'm just making this up. I'm saying the verbatim title rule is a poor fit here because it took a relevant (central, even!) part of the headline and moved it to a spot of secondary importance and size. There are cases where I defend the rule, but right now I am talking about this case and only this case :)
  - Dylan16807 7y ago
    So even if the url got bumped to 12pt, you'd complain if the rest was 15? I think that's weird. As long as it's on the same line as the title and easily legible, I really don't see a problem. And it's not a spot of secondary importance. If it was still in the title, making it longer, the spot where you see the url would have title in it.
    - pbhjpbhj 7y ago

maxpert 7y ago

I’ve used LZ4 and Snappy in production for compressing cache/mq payloads. This is on a service serving billions of clicks in a day. So far very happy with the results, I know zstd requires more CPU than LZ4 or snappy on average but has someone used it under heavy traffic loads on web services. I am really interested trying it out but at the same time held back by “don’t fix it if it ain’t broken”.

loeg 7y ago
Zstd has "fast" negative levels (-5, -4, ... -1, 1, ..., 22). -4 or -5 are purportedly comparable (but not quite as good) as LZ4.
- ncmncm 7y ago
  Better to just use Lz4, then.
  - emn13 7y ago
    Maybe. The thing is; zstd is quite close, and unlike lz4, zstd has a broad curve of supported speed/time tradeoffs. Unless you're huge and engineering effort is essentially free or at least the microoptimization for one specific ratio is worth the tradeoff - you may be better off choosing the solution that's less opinionated about the settings. If it then turns out that you care mostly about decompression speed + compression ratio and a little less about compression speed, it's trivial to go there. Or maybe it turns out you only sometimes need the speed, but usually can afford spending a little more CPU time - so you default to higher compression ratios, but under load use lower ones (there's even a streaming mode built-in that does this for you for large streams). Or maybe your dataset is friendly to the parallization options, and zstd actually outperforms lz4. If you know your use case well and are sure the situation won't change (or don't mind swapping compression algorithms when they do), then lz4 still has a solid niche, especially where compression speed matters more than decompression speed. But in many if not most cases I'd say it's probably a kind of premature optimization at this point, even if you think you're close to lz4's sweet spot.
7y ago

kbumsik 7y ago

> Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup. Impressive. As a AUR package maintainer I am also wondering how the compression speed is though.

ncmncm 7y ago
Compression speed is many, many, many times faster than xz, and (only) much faster than gzip. Really, only lz4 beats it.
- integricho 7y ago
  After reading these comments, I can't help but wonder, what is the benefit of Zstd over lz4? Why didn't they switch to lz4 if it was the speed of the algorithm that they favored even with marginally worse compression ratios?
  - isatty 7y ago
    Guessing that 0.8x size increase for 1300% speedup was worth the tradeoff but maybe ≥1.5 size increase or more was not (especially considering a 1300%->2000% increase is not going to be user visible for 99% of the packages).
    - TJSomething 7y ago
      It's not 0.8 times size increase, it's a 0.008 times size increase, since the unit is percent. The latter seems pretty marginal to me.
  - ncmncm 7y ago
    Where Zstd will reduce, say, 3x, Lz4 reduces only 2.5x. This doesn't seem very different until you look at it from the other end: my .zst file is 3.3 GB, but the .lz4 would have been 4 GB, which is 700 MB more. Was a time when 700 MB mattered; it was as much as you could get onto a CD. So, there is a place for each. I would set up the process to use Lz4 when testing, and Zstd for actual delivery to download archives. In some circumstances, particularly when using a shared file server, Lz4 can be quite a lot faster than writing and reading data uncompressed.

loeg 7y ago

I'd love to see Zstandard accepted in other places where the current option is only the venerable zlib. E.g., git packing, ssh -C. It's got more breadth and is better (ratio / cpu) than zlib at every point in the curve where zlib even participates.

ncmncm 7y ago
Wireshark! Wireshark! Also lz4, of course.
jacobolus 7y ago
It would be great to see better compression supported by browsers.
- tyingq 7y ago
  Chrome apparently tested zstd out, and it's an improvement over Chrome's forked and optimized zlib on x64, but slower on ARM/Android. https://bugs.chromium.org/p/chromium/issues/detail?id=912902#c6 https://bugs.chromium.org/p/chromium/issues/detail?id=912902... Few (or none?) of Chrome's fairly dramatic improvements to zlib have been upstreamed. https://github.com/madler/zlib/issues/346 https://github.com/madler/zlib/issues/346 Edit: Also, if browsers do adopt zstd, it's likely you'll end up with the same situation where they fork their own implementation of zstd. Upstreaming requires signing Facebook's CLA, which has patent clauses that don't work for most.
- dictum 7y ago
  Brotli has wide browser support (https://caniuse.com/#feat=brotli https://caniuse.com/#feat=brotli) and comes closer to zstd in compression ratio and compression speed, but its decompression speed is significantly lower and closer to zlib. https://github.com/facebook/zstd#benchmarks https://github.com/facebook/zstd#benchmarks AFAIK (I haven't looked much into it since 2018) it's not widely supported by CDNs, but at least Cloudflare seems to serve it by default (EDIT: must be enabled per-site https://support.cloudflare.com/hc/en-us/articles/200168396-What-will-Cloudflare-compress- https://support.cloudflare.com/hc/en-us/articles/200168396-W...)

Annatar 7y ago

I couldn't care less about decompression speed, because the bottleneck is the network, which means that I want my packages as small as possible. Smaller packages mean faster installation; at 54 MB/s or faster decompression rate of xz, I couldn't care less about a few milliseconds saved during decompression. For me, this decision is dumbass stupid.

ubercow13 7y ago
Why do you care so much about the few extra miliseconds wasted downloading, then? (0.8% size increase is ~ 0). Also don't forget that Arch can also be used on machines with very slow CPU but very fast network connections, like many VPSs. I think this will make a tangible difference on mine. This is also a big improvement for package maintainers and anyone building their own packages without bothering to modify the makepkg defaults, eg. most people using an AUR helper.
- Annatar 7y ago
  Because size does matter.
snvzz 7y ago
The extra decompression complexity might be a joke on a Zen2 server, but it definitely is not in older systems. If this was netbsd m68k, you'd probably easily understand.
- Annatar 7y ago
  I use xz on my A1200 all of the time, and Amiga is the stereotypical system where maximum possible compression matters over everything else. Don't make assumptions about me.
  - snvzz 7y ago
    I applaud your patience. Even with my vampire, I'll use something faster whether at all possible. May I ask, why xz over, say, PAQ8PF?

Phlogi 7y ago

The wiki is already up to date if you build your own or AUR packages and want to use multiple cpu cores https://wiki.archlinux.org/index.php/Makepkg#Utilizing_multiple_cores_on_compression https://wiki.archlinux.org/index.php/Makepkg#Utilizing_multi...

rwmj 7y ago

I wish zstd supported seeking and partial decompression (https://github.com/facebook/zstd/issues/395#issuecomment-535875379 https://github.com/facebook/zstd/issues/395#issuecomment-535...). We could then use it for hosting disk images as it would be a lot faster than xz which we currently use.

ncmncm 7y ago
Fun fact: Two zstd files appended is a zstd file. Also, parallel zstd must have some way to split up the work, that you could maybe use too.
- rwmj 7y ago
  I would suggest reading the github issue that I linked to, you'll see why it's not currently possible.

vmchale 7y ago

What of lzip?

gravitas 7y ago

AUR users -- the default settings in /etc/makepkpg.conf (delivered by the pacman package as of 5.2.1-1) are still at xz, you must manually edit your local config: PKGEXT='.pkg.tar.zst' The largest package I always wait on perfect for this scenario is `google-cloud-sdk` (the re-compression is a killer -- `zoom` is another one in AUR that's a beast) so I used it as a test on my laptop here in "real world conditions" (browsers running, music playing, etc.). It's an old Dell m4600 (i7-2760QM, rotating disk), nothing special. What matters is using default xz, compression takes twice as long and appears to drive the CPU harder. Using xz my fans always kick in for a bit (normal behaviour), testing zst here did not kick the fans on the same way. After warming up all my caches with a few pre-builds to try and keep it fair by reducing disk I/O, here's a sampling of the results: xz defaults - Size: 33649964 real 2m23.016s user 1m49.340s sys 0m35.132s ---- zst defaults - Size: 47521947 real 1m5.904s user 0m30.971s sys 0m34.021s ---- zst mpthread - Size: 47521114 real 1m3.943s user 0m30.905s sys 0m33.355s I can re-run them and get a pretty consistent return (so that's good, we're "fair" to a degree); there's disk activity building this package (seds, etc.) so it's not pure compression only. It's a scenario I live every time this AUR package (google-cloud-sdk) is refreshed and we get to upgrade. Trying to stick with real world, not synthetic benchmarks. :) I did not seem to notice any appreciable difference in adding the `--threads=0` to `COMPRESSZST=` (from the Arch wiki), they both consistently gave me right around what you see above. This was compression only testing which is where my wait time is when upgrading these packages, huge improvement with zst seen here...

Foxboron 7y ago
It should be noted that the makepkg.conf file distributed with pacman does not contain the same compression settings as the one used to build official packages. pacman: COMPRESSZST=(zstd -c -z -q -) https://git.archlinux.org/svntogit/packages.git/tree/trunk/makepkg.conf?h=packages/pacman#n133 https://git.archlinux.org/svntogit/packages.git/tree/trunk/m... devtools: COMPRESSZST=(zstd -c -T0 --ultra -20 -) https://github.com/archlinux/devtools/blob/master/makepkg-x86_64.conf#L135 https://github.com/archlinux/devtools/blob/master/makepkg-x8...
- gravitas 7y ago

m4rtink 7y ago

BTW, Fedora recently switched to zstd compression for its packages as well. For the same resons basically - much better overall de/compression speed while keeping the result mostly the same size. Also one more benefit of zstd compression, that is not widely noted - a zstd file conpressed with multiple threads is binary the same as file compressed with single thread. So you can use multi threaded compression and you will end up with the same file cheksum, which is very important for package signing. On the other hand xz, which has been used before, produces a binary different file if compressed by single or multiple threads. This basucally precludes multi threaded compression at package build time, as the compressed file checksums would not match if the package was rebuild with a different number of compression threads. (the unpacked payload will be always the same, but the compressed xz file will be binary different)

imtringued 7y ago

This blog post probably wasted more of my time than I will ever gain from the faster decompression...

JeremyNT 7y ago

I learned about this one the hard way when I went to update a really crufty (~ 1 year since last update) Arch system I use infrequently the other day. I had failed to update my libarchive version prior to the change and the package manager could not process the new format. Luckily updating libarchive manually with an intermediate version resolved my issue and everything proceeded fine. This is a good change, but it's a reminder to pay attention to the Arch Linux news feed, because every now and then something important will change. The maintainers provided ample warning about this change there (and indeed I had updated by other systems in response) so we procrastinators really had no excuse :)

zerogara 7y ago

Most of the results published show very little positive or negative speed in decompression, where is all this -1300% coming from? edit: Sorry, my fault that was decompression RAM I was thinking about, not speed, although I was influenced by my test that without measuring both xz and zstd seemed instant.