5 ms·
OK, I'll bite. This article is overblown in its criticisms and has some issues of its own. 1. The article itself recommends that "a tool is supposed to do one
by jl6 10y ago
OK, I'll bite. This article is overblown in its criticisms and has some issues of its own.
1. The article itself recommends that "a tool is supposed to do one thing and do it well". Well, xz is for compression, nothing else. If you want to check integrity, use a hashing tool such as sha256sum. If you want to recover from errors, generate parity files using par2 or zfec.
2. The article claims xz was chosen by prominent open source projects due to hype. No, it was chosen because of its favorable size/speed tradeoffs.
3. It's not clear what the author means by "long-term archiving". Archives professionals (as in, people who are actually employed by memory institutions) will tell you of many factors (all unmentioned here) that have a bearing on whether a file format (particularly a compressed one) is suitable for use in data preservation.
4. The section on trailing data is particularly bizarre. By claiming that xz is "telling you what you can't do with your files", it seems that the author considers it perfectly reasonable to append arbitrary data to the end of a file and expect it to continue functioning. Is my normality detector off today or is this just a wacky thing to want to do?
5. The article is written by the author of lzip (which it compares favorably to xz), but does not disclose this. Admittedly it is hosted on the lzip site, but overall the article comes across as an opinionated hit-piece designed to sow doubt in a competitor.
- LukeShu 10y agoRegarding #4: Padding out a file is popular because of tape drives. You'll typically write the archive to the an appropriately sized tape raw, without a filesystem (which is just another format you have to maintain a parser for). So because the file will be smaller than the tape, there will be garbage at the end of the tape. Because we wrote the file to the tape directly, there is no way of knowing where it ends, and a tool reading the file will need to just deal with the garbage data.
- Spooky23 10y agoOld man moment! Kids these days don't appreciate having random addressable storage for archive/backup data!
- the_mitsuhiko 10y agoGarbage data is not a problem since the length is known.
- LukeShu 10y agoI do not understand the xz format enough to evaluate that claim myself, but TFA explicitly claims that garbage data is a problem.
- marcosdumay 10y agoHum, no. When you tar a set of files directly into a tape, you don't know the resulting tar size beforehand. Even less if you compress the result.
- mnarayan01 10y agoI would think you could just write the size at the end of the tape?
- ikawe 10y agoWhen you're reading the data later, how would you know where to find "the end" if you don't already know the length?
- mnarayan01 10y agoYou don't know the end of the data, but presumably(?) you know the end of the tape.
- marcosdumay 10y agoWe are talking past each other here... Making a backup with tar is done by typing something like that on bash: > tar -c - dir1 dir2 dir3 > /dev/tape That will (hopefully, I doubt I got the tar switches right) backup those dirs into the tape (that will actually have a weird name, not '/dev/tape'). Now, in practice Linux doesn't always know the size of a tape you inserted. But this is not the issue, if you accept the seeks needed for that, you'd better write at the beginning anyway.
- jl6 10y agoDoesn't one typically write to tape drives with tar, rather than writing a raw file?
- LukeShu 10y agoIndeed. This is why the tar format explicitly allows garbage data at the end. So then people started pondering all of the nifty or clever things they could do with tar files. And they didn't want to give it up when they started compressing the tar files.
- zeta0134 10y agoCouldn't you just reverse the order then, and create an xz.tar? Maybe I don't understand the benefit of taring the data first.
- wojcech 10y agoTar is just bundling into a single file AFAIK. There is a slight benefit depending on your compression tool to tar and then compress, because (AFAIK again)some tools compress files individually and then write them into a hierarchical file(I guess this is what xz does as well, since it's searchable?). If you tar first, these tools will work better, since they encode patterns found in all files instead if doing it per file(which means e.g. if there is a header once per file, that will get compressed in the tar.comorpress, not in the . compress)
- baruch 10y agoI've created sortedtar Brewster if this assumption and while it is correct tree benefit is mostly negligible except for some edge cases.
- kahrkunne 10y agoConsidering neither lzip nor xz is a commercial product, I don't think your final point stands. To me, it only makes sense that the person who wrote a "competing" file format would have a strong negative opinion of xz
- saurik 10y ago> The article claims xz was chosen by prominent open source projects due to hype. No, it was chosen because of its favorable size/speed tradeoffs. No: that's why the LZMA compression format was chosen, and why the .lzma container format temporarily became popular as people started using lzma-tools. The move to xz, which results in slightly larger files for seemingly arbitrary reasons and which technically provides one benefit (seekability) almost everyone defeats (by compressing a tar file) while using the same relying on the same underlying compression algorithm is less-well motivated.
- jl6 10y agoWere there many prominent open source projects using the LZMA format before the creation of xz? Debian, for example (which is where this particular flamefest originates), adopted[0] xz the year following the stable v5.0.0 of XZ-Utils[1]. My impression was that the whole 7zip->p7zip->lzma_alone->xz journey was concluded before any "xz hype" began. [0] https://lists.debian.org/debian-devel-announce/2011/08/msg00001.html https://lists.debian.org/debian-devel-announce/2011/08/msg00... [1] http://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD http://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD
- saurik 10y agoI don't quite understand the timeline you are drawing here but the Debian Package Manager is a perfect example of a prominent open source project which used the LZMA format before the creation of xz. They recently shifted to "deprecate" the LZMA format, a change which I have removed from the negligible fork of dpkg I ship to the tens of millions of people running Cydia.