11 ms·
Xz
- spicyj 16y agoCan someone explain to me why this Wikipedia article is at the top of the front page?
- lanstein 16y agoI'm guessing because people voted it up. Additionally, it's not a peak time right now.
- mechanician 16y agoWhen is peak time?
- deleted 16y ago[deleted]
- wmf 16y agoMost hackers don't know about XZ yet, but they should. I still prefer not to see Wikipedia articles on HN since they lack context. A "7 reasons you should ditch bzip2 and use XZ" blog post would probably be better.
- someone_here 16y agoOh god please not more "X reasons for Y"
- WilliamLP 16y agoGive me Z reasons why you don't like "X reasons for Y" posts.
- SkyMarshal 16y ago9 Reasons Why You Should Hate "X Reasons for Y" posts: 1. They're overdone 2. They're uncreative 3. They're unoriginal 4. They're a crutch 5. They're assembly-line writing 6. They're used by tabloids to appeal to supermarket zombies 7. They've helped turn respectable mags into tabloids (or are a symptom of it, not sure which came first) 8. They're psychologically manipulative (I don't know how, I just know it) 9. They work :)
- joshfinnie 16y agoYou forgot to put these across 11 individual comments so your 8pts would have been multiplied to 88! I do not mind lists, but I cannot stand lists that are seperated to pages for inflated page counts!
- koeselitz 16y agoHe was going to put it across 11 comments, but as the Wikipedia article clearly points out, xz compression conventionally bundles a single file.
- deleted 16y ago[deleted]
- Perceval 16y agoThe HN Guidelines explicitly frown upon "X reasons..." titles because they're almost always linkbait. If the original title begins with a number or number + gratuitous adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids." http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html
- deleted 16y ago[deleted]
- snippyhollow 16y ago"X reasons for Z": there, I fixed it for you.
- tvon 16y agoI'd prefer it to a two paragraph Wikipedia article.
- ramchip 16y agoHow about a "Y is the new X" or "X is dead, long live Y"?
- ElliotH 16y agoAgree on the wikipedia articles with no context point. Especially since the article in question is short with not very much real information on why it matters.
- matthavener 16y agoLZMA2 has a better compression ratio than bzip2, according to this article http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm http://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Mark... . Seems like it would be a good idea to switch away from bzip2/gzip to zx
- wheels 16y agoThe main reason to continue to use gzip is speed: it's an order of magnitude faster than bzip2 for compression / decompression, for a relatively small compression ratio hit. It's fast enough to us inline in web servers, SSH sessions, etc, and if you're just moving data around a single time, often the amount of time for compress / transfer / decompress is much faster than with bzip2.
- alecco 16y agoYes, the new compression algorithms (BWT and LZ77-based) are slower and require a lot more memory than DEFLATE (*zip) in most cases. This guy did a not so rigorous analysis but it is mostly OK: http://changelog.complete.org/archives/931-how-to-think-about-compression-part-2 http://changelog.complete.org/archives/931-how-to-think-abou... A little bird told me there's a better algorithm in speed/compression rate coming soon ;)
- gxti 16y agoI've found that on many types of data, xz -2 is twice as fast and compresses better than gzip -9. Of course gzip -1 kicks the pants off xz but there's absolutely no reason to use gzip at its higher levels, nor to use bzip2 at all. And if you want fast, LZO does it better -- the lzop tool provides a typical unix compress interface. Using 50MiB from a tarball I had lying around, best of 3: prog time(s) size(%) gzip -1 1.703 38.7 gzip -9 14.670 35.1 (worse than xz -1) bzip2 -1 7.714 34.0 bzip2 -9 8.035 31.4 (worse than xz -1) xz -1 6.278 31.3 xz -9 42.445 22.7 (best compression) lzop -1 0.670 46.1 (fastest) lzop -9 18.144 38.0
- 16y ago
- kiba 16y agoIt's apparently something new that has been floating around. Some hackers are probably curious about the "new thing" that popped up lately.
- scdlbx 16y agoSoftware that was released 9 months ago isn't particularly new.
- joshfinnie 16y agoFirst, I think that is why it's in quotes. Second, I am reading Of Mice or Men for the first time this summer. It is new to me!
- golwengaud 16y agoIn retrospect, I probably should have submitted the XZ Utils project page: http://tukaani.org/xz/ http://tukaani.org/xz/ . However, that page does not give the same picture of widespread adoption that the Wikipedia article does. That use was what interested me: I'm surprised that it only came to my attention when I was trying to recall the name of the 7z command line utility p7zip.
- goodside 16y agoI upvoted it because, on top of being interesting, it fights the passively accepted wisdom that "newsworthy" is synonymous with "worthy".
- tman 16y agoXz appears to use the same algorithm as 7zip. Here are some comparisons between the big 3 compression algorithms (taken from http://blogs.reucon.com/srt/tags/compression/ http://blogs.reucon.com/srt/tags/compression/ -- he used a 163 MB Mysql dump file for the tests): Compressor Size Ratio Compression Decompression gzip 89 MB 54 % 0m 13s 0m 05s bzip2 81 MB 49 % 1m 30s 0m 20s 7-zip 61 MB 37 % 1m 48s 0m 11s
- swombat 16y agoSo bzip2 and 7-zip are way, way slower than gzip, then? Bandwidth is cheap. Stick to gzip.
- wmf 16y agoGiven the increasing CPU/network gap, it's only a matter of time before the bandwidth (and thus time) saved by XZ more than compensates for the slower compression.
- bfung 16y agomileage may vary; If the time it takes to compress is costs much less than the time it takes to transfer over the network, you might want not want to use gzip. For example, you're transferring a large file (1GB? 1TB?) to a remote person to deal with, is it cheaper to gzip (lower compression rate), take longer for the network transfer (most likely slowest step), and have the other person unzip, or to use a better compressor, and have the file transferred over quicker?
- w1ntermute 16y agoSo bzip2 and 7-zip are way, way slower than gzip, then? Bandwidth is cheap. Stick to gzip. It's not as simple as that. Which one is better depends on the use case. If you're sending a one-off file to somebody, sure, gzip is better. But if you want to distribute a file to a large number of people (like Linux distributions do with their packages), the extra CPU time is insignificant compared to the bandwidth saved over the course of thousands of downloads.
- marcinw 16y agoThe only reason I stick with gzip over bz2 or 7z (lzma) is that gzip is everywhere. In reality, nobody has a file compression utility that can handle LZMA (even though I do, I never use it).
- swolchok 16y agoInterestingly, file(1) can't identify lzma(1)-compressed files (they show up as data), but unlzma(1) reassuringly knows the difference between LZMA and random data. This came up once or twice in DEFCON CTF quals (http://www.vnsecurity.net/2010/05/defcon-18-quals-writeups-collection/ http://www.vnsecurity.net/2010/05/defcon-18-quals-writeups-c...).
- wmf 16y agoThat's why you should use xz instead of lzma (they're not the same thing) — the XZ format has a magic number.
- dbz 16y ago7-Zip utility (for compressing and decompressing different file types) doesn't?
- VMG 16y agoArch Linux already switched to xz as the default compression for packages
- acg 16y agoThe specification ( http://tukaani.org/xz/xz-file-format.txt http://tukaani.org/xz/xz-file-format.txt ) reads like this format allows the application of a chain of compression "filters". Perhaps the most interesting, and least portable is the filter that modifies code to make compression easier. (Section 5.2.3) Is this mainly for intel architecures?
- gubatron 16y agoTried to apt-get it (on Ubuntu 9.10) but got scared when I gotta a warning about lzma getting uninstalled and possibly breaking my dpkg. Looking good on ubuntu 10.4 Same folder compressed in 1/2 the size as with tar cvfz, sick.
- ars 16y agoxz is backward compatible with lzma. It can both create and decompress lzma archives, and it has a tool with the same command line options. lzma is depreciated by its author in favor of xz.
- kunley 16y agoxz-utils deprecates lzma. $ dpkg -s xz-utils|grep Replaces Replaces: lzma $ dpkg -S $(which lzma) xz-utils: /usr/bin/lzma
- vasi 16y agoI wrote a tool, pixz, that does xz compression in parallel to take advantage of multiple cores. It also indexes xz-compressed tarballs, so you can extract an individual file very quickly instead of needing to decompress the entire tarball. The parallel-compressed, self-contained tarball+index is fully compatible with regular tar/xz, you don't need any special tools to extract it. http://github.com/vasi/pixz http://github.com/vasi/pixz The interface is still very rough, but it works. The xz utility comes with a very nice library and API, which made this a lot easier--thanks, Lasse!
- coderdude 16y agoWhile we're talking about compression algorithms, here is a nice little gem called BMZ: http://bitbucket.org/mattsta/bmz/src http://bitbucket.org/mattsta/bmz/src It's is a fast compression scheme implemented using BMDiff and a Google Zippy clone (based off LZO).
- deleted 16y ago[deleted]
- dschoon 16y agoI built and tried this out. I'm rather shocked to see speed and compression ratio better than gzip -9. The results of my unscientific test (compression only): Compressor Size Ratio Time gzip -1 23MB 88% 1.18s gzip -2 23MB 87% 1.38s bzip2 23MB 87% 5.57s xz -1 23MB 87% 5.35s xz -9 11MB 43% 10.58s bmz 13MB 45% 0.95s
- vicaya 16y agoIt really depends on the data. For highly redundant data like web pages with lots of boilerplate header/footer, it can compress better because the bm_pack (first pass of bmz) looks for large common patterns over all the input. For typical text, it should be a little worse than gzip but faster. BMZ = bmpack + lzo by default and can be combined with lzma if necessary. It's not really a BMDiff and Zippy clone, as I've never had a chance to see Google's implementation. It's based on the original Bentley & McIlroy paper: "Data Compression Using Long Common Strings", 1999. Even the two pass idea is from that paper. It was really a wacky experimental implementation (with a lot of room for improvement) to satisfy my curiosity. I'm a little surprised that the 0.1 version has been stable for quite a few people compressing TBs of data through it.
- dmn001 16y agoI use 7-zip 9.15 beta - it supports LZMA2 compression and available to download here: http://www.7-zip.org/ http://www.7-zip.org/
- india 16y agoAlso notably, slackware has shifted it's packaging format to a xz tar since the last two releases. This validates xz as a good replacement for gz/bz2.