15 ms·

Xz format inadequate for long-term archiving (2016)

92 points by UkiahSmith 7y ago

brianpgordon 7y ago
Previous discussions: https://news.ycombinator.com/item?id=12768425 https://news.ycombinator.com/item?id=12768425 https://news.ycombinator.com/item?id=16884832 https://news.ycombinator.com/item?id=16884832
LinuxBender 7y ago
If you first use tar to preserve xattrs/etc.. then you can use anything to compress. xz, bz2, 7z, even arj if you are feeling nostalgic. tar cvfJ ./files.tar.xz /some/dir
- zamalek 7y ago
  You've missed the point of the article entirely. A single bit-flip (which is almost guaranteed over long-term) can easily render the entire xz file corrupt. This has nothing to do with xattrs/etc.
  - LinuxBender 7y ago
    Yes, I am totally on auto-pilot today. I'm used to a different article that gets re-posted often about xz and my browser blocks non-https sites so I assumed it was that other article. That said, I use xz in automation that compresses files on one end and decompresses on the other. I've not had any file corruption thus far. checksums always match. Hopefully the author has submitted bug reports and ways to reproduce.
  - imiric 7y ago
    I've been using tar+pixz+par2 for backups for a while now, but this article still worries me. What can I replace pixz with that compresses as well and keeps the indexing functionality? I'd like to avoid zstd because Facebook.
SEJeff 7y ago
This should have (2016) in the title.
- Thoreandan 7y ago
  and "from the author of lzip, a competing lzma library that never went viral". Welcome to the Better Technology that Shoulda Made It bench. Your seat's over there next to OS/2, BeOS, and OpenGenera.
  - nkoren 7y ago
    Amiga forever!!!!!!
    - zmix 7y ago
      Ha! I just wanted to add this. But you did it first! :-)
microcolonel 7y ago
> "3 Then, why some free software projects use xz?" Because the files are usually smaller than gzip, with faster decompression than bzip2, and the library is available on most systems.
- h1d 7y ago
  Archiving for distribution and backups are very different things. You don't care if some app distribution compressed file gets corrupted, you just compress again but your compressed backup files usually don't have much source of reference. I wouldn't use any unreliable format for backups. I picked bzip2 for stability and compression rate.
  - microcolonel 7y ago
    In my opinion, the compressor is not the right place to add data integrity mechanisms, especially since data integrity mechanisms only really apply to particular media. Data on hard drives don't get corrupted in the same way as data on TLC SSDs, and generally on the latter you're better off with redundancy and diversification, than with inline error correcting codes. Honestly, I don't see why xz should have any of its own data integrity mechanisms whatsoever, except maybe a whole-archive CRC32 or similar.
    - duskwuff 7y ago
      Right. The purpose of an archiver/compressor is to store a bunch of files together, and use as little space as possible to do it. Data integrity / error correction / redundancy all lie in the opposite direction of that goal.
Adamantcheese 7y ago
How about something like ZPAQ instead for archiving? Especially if you're doing backups and not a lot of the information is changing.
jordigh 7y ago
This guy used to go around GNU mailing lists (and others) trying to get us to use lzip. https://gcc.gnu.org/ml/gcc/2017-06/msg00044.html https://gcc.gnu.org/ml/gcc/2017-06/msg00044.html https://lists.debian.org/debian-devel/2017/06/msg00433.html https://lists.debian.org/debian-devel/2017/06/msg00433.html It was a bit bizarre when he hit the Octave mailing list. Eventually, people just wanted xz back: http://octave.1599824.n4.nabble.com/opinion-bring-back-Octave-xz-source-release-td4683705.html http://octave.1599824.n4.nabble.com/opinion-bring-back-Octav...
- AdmiralAsshat 7y ago
  Everyone's gotta have their white whale, I guess.
esaym 7y ago
Interestingly, since "recovery" is mentioned several times, I decided to test myself. I took a copy of a jpeg image, compressed it different times with either gzip or bzip2, then with a hexeditor modified one byte. The recovery instructions for gzip is to simply do "zcat corrupt_file.gz > corrupt_file". While for bzip2 is to use the bzip2recover command which just dumps the blocks out individually (corrupt ones and all). Uncompressing the corrupt gzip jpeg file via zcat at all times resulted in an image file the same size as the original and could be opened with any image viewer although the colors were clearly off. I never could recover the image compressed with bzip2. Trying to extract all the recovered blocks made by bzip2recover via bzcat would just choke on the single corrupted block. And the smallest you can make a block is 100K (vs 32K for gzip?). Obviously pulling 100K out of a jpeg will not work. Though I'm still confused as to how the corrupted gzip file extracted to a file of the same size as the original. I guess gzip writes out the corrupted data as well instead of choking on it? I guess gzip is the winner here. Having a file with a corrupted byte is much better than having a file with 100K of data missing...
- wereHamster 7y ago
  Whether recovery leads to (almost) useable data depends on what byte you modify. It's entirely possible that a single corrupt byte in the compressed data leads to a single corrupt byte when uncompressed. When you are dealing with images you may not even notice that a single pixel is wrong. But it's also possible that you completely destroy the data such that the decompression algorithm can't even deal with it and has to give up.
  - chasil 7y ago
    A decade and a half ago, I wrote an Oracle archived log that I had compressed with bzip2 to a DLT40 tape. I recovered and uncompressed (without error) the log, then tried to apply it to a database recovery which rejected it as corrupt. After several attempts to read the tape (amounting to dozens of hours), I finally put it in the original drive that wrote it and pulled the file to the remote recovery system - this worked. I immediately began including PAR2 files on the tapes, so the restored contents could be verified and corrected. I have my doubts that bzip2 is as sensitive to corruption as the author of asserts, but perhaps there have been improvements to the code since my misfortune.
asveikau 7y ago
I remember seeing this article before. This time the reaction that surges for me is: if you want long-term archiving but don't assume redundant storage, it's not going to go well. Put your long-term archives on ZFS.
- StavrosK 7y ago
  Why are you assuming they aren't assuming redundant storage? Redundant storage isn't a cure-all, there's still a chance two blocks on two disks will fail in the exact same spot.
  - kipari 7y ago
    I reckon that the chance of the same two blocks on two different disks failing between ZFS scrubs would be incredibly small.
    - StavrosK 7y ago
      Yes, over normal timescales. A lot can happen in a thousand years.
      - asveikau 7y ago
        Thousands of years is a lot of scrubs and a lot of disk replacements, though. And a solution like ZFS, properly monitored, should help make those detections and repairs happen early, with lower odds of loss. Although honestly in a thousand year timeframe I very much doubt humanity will preserve ZFS, gzip, tar, jpeg, PNG, ASCII, today's spoken and written languages in current form, etc. Just as written material from 1000 years ago is not very accessible to most people; with the original material you need intense study before you even know what you're looking at.
    - cmurf 7y ago
      It is incredibly small if you don't consider either drive failing. But if one drive fails, it happens with some regularity that a sector on the good drive is bad. In actuality, only one sector is bad, but in effect the dead drive means its mirror is also bad. This comes up on the linux raid list with some frequency whenever there are drive failures with raid56, and the subsequently the raid trips over a single bad sector. But it's true that lack of scrubbing contributes to this scenario, as well as the terrible combination of consumer drives with very high bad sector recovery times and the Linux SCSI command timer default of 30 seconds. That combination ends up causing a masking of bad sectors that end up not getting repaired, and as a user you may not realize that the link resets are not normal and suggest a bad sector as the cause.
xoa 7y ago
Not that many of the complaints aren't reasonable, but I thought that in general compression/format was orthogonal to parity, which is what I assume is actually wanted for long-term archiving? I always figured that the goal should normally to be able to get back out a bit-perfect copy of whatever went in, using something like Parchive at the file level or ZFS for online storage at the fs level. I guess on the principle of layers and graceful failure modes it's better if even sub-archives can handle some level of corruption without total failure, and from a long term perspective of implementation independence simpler/better specified is preferable, but that still doesn't seem to substitute for just having enough parity built in to both notice corruption and fully recover from it to fairly extreme levels.
- matt-attack 7y ago
  I think with archiving it’s more than that. Sure you can guarantee that the actual tool you just compressed with can restore the original perfectly. But with long term digital archiving I think you need the assurance that the “spec” called “xz” could be perfectly reimplemented by an expert in the future. Based solely on documentation. And on a platform that doesn’t exist today. That is, you must assume the original executable is either not available or not able to be executed.
  - xoa 7y ago
    I don't disagree with that at all! I did try to say (maybe unclearly) that having simpler more foolproof and failsafe layers at every level seems absolutely worth pursuing anyway, where possible. But I also wonder whether some of the common wisdom is from an age that is obsolete for many scenarios? Ie., in the 80s and 90s and even early 00s there was a lot more churn, practices were less standardized, computing time was more expensive, storage capacity was far more expensive, the ratio of software to data size was higher, etc. The latter seems to tie into "must assume the original executable is either not available or not able to be executed." For serious archiving, does it no make sense to just bundle in not merely the executable, but in some instances an entire environment? In a "fall of civilization" type scenario that may not be helpful vs a clear simple spec and ease of bootstrapping, but for situations where technological continuity is a limiting factor for some reason anyway is it safe to simply assume basics like at this point x86 will never go away as something that is at least virtualized? In my own experience there is a pretty clear cutoff date after which I can continue to run the entire environment in a VM. Again this is shooting the breeze a bit, article is discussing a case where there should be the freedom to choose better formats. But for a lot of important archive material, including software itself, are we getting to the point where many long term archives should simply including everything necessary to deal with them in the present day as a container or VM image, which is then stored with a solid amount of parity and replication?
shmerl 7y ago
xz-utils should implement parallel decompression already. pixz is doing it, but stock xz is not. Most end users benefit from faster decompression.
ltbarcly3 7y ago
No file format is perfect, I've been using xz for years and I can't think of a single issue I have had. The compression rate is dramatically better than gzip or bzip2 for many types of archives (especially when there is a large redundancy, for example when compressing spidered web pages from the same site you can get well over 99% size reduction compared to 70% reduction for gzip, which means using less than one 30th of the disk space). Lately I have been using zstd for some things since it gives good compression and is much faster than xz. This criticism of xz just seems nit picky and impractical, especially if you are compressing tar archives and/or storing the archives on some kind of raid which can correct some read errors (such as raid5).
mkj 7y ago
A bit of speculation here, but perhaps xz won over lzip because it has a real manpage? lzip has the usual infuriating short summary of options with a "run info lzip for the complete manual". Also the source code repository doesn't even seem linked directly from the lzip homepage - technical considerations aren't the only thing that determines if software is "better", it also has to be well presented.

92 points by UkiahSmith 7y ago

Previous discussions: https://news.ycombinator.com/item?id=12768425 https://news.ycombinator.com/item?id=12768425 https://news.ycombinator.com/item?id=16884832 https://news.ycombinator.com/item?id=16884832

LinuxBender 7y ago

If you first use tar to preserve xattrs/etc.. then you can use anything to compress. xz, bz2, 7z, even arj if you are feeling nostalgic. tar cvfJ ./files.tar.xz /some/dir

zamalek 7y ago
You've missed the point of the article entirely. A single bit-flip (which is almost guaranteed over long-term) can easily render the entire xz file corrupt. This has nothing to do with xattrs/etc.
- LinuxBender 7y ago
  Yes, I am totally on auto-pilot today. I'm used to a different article that gets re-posted often about xz and my browser blocks non-https sites so I assumed it was that other article. That said, I use xz in automation that compresses files on one end and decompresses on the other. I've not had any file corruption thus far. checksums always match. Hopefully the author has submitted bug reports and ways to reproduce.
- imiric 7y ago
  I've been using tar+pixz+par2 for backups for a while now, but this article still worries me. What can I replace pixz with that compresses as well and keeps the indexing functionality? I'd like to avoid zstd because Facebook.

SEJeff 7y ago

This should have (2016) in the title.

Thoreandan 7y ago
and "from the author of lzip, a competing lzma library that never went viral". Welcome to the Better Technology that Shoulda Made It bench. Your seat's over there next to OS/2, BeOS, and OpenGenera.
- nkoren 7y ago
  Amiga forever!!!!!!
  - zmix 7y ago
    Ha! I just wanted to add this. But you did it first! :-)

microcolonel 7y ago

> "3 Then, why some free software projects use xz?" Because the files are usually smaller than gzip, with faster decompression than bzip2, and the library is available on most systems.

h1d 7y ago
Archiving for distribution and backups are very different things. You don't care if some app distribution compressed file gets corrupted, you just compress again but your compressed backup files usually don't have much source of reference. I wouldn't use any unreliable format for backups. I picked bzip2 for stability and compression rate.
- microcolonel 7y ago
  In my opinion, the compressor is not the right place to add data integrity mechanisms, especially since data integrity mechanisms only really apply to particular media. Data on hard drives don't get corrupted in the same way as data on TLC SSDs, and generally on the latter you're better off with redundancy and diversification, than with inline error correcting codes. Honestly, I don't see why xz should have any of its own data integrity mechanisms whatsoever, except maybe a whole-archive CRC32 or similar.
  - duskwuff 7y ago
    Right. The purpose of an archiver/compressor is to store a bunch of files together, and use as little space as possible to do it. Data integrity / error correction / redundancy all lie in the opposite direction of that goal.

Adamantcheese 7y ago

How about something like ZPAQ instead for archiving? Especially if you're doing backups and not a lot of the information is changing.

jordigh 7y ago

This guy used to go around GNU mailing lists (and others) trying to get us to use lzip. https://gcc.gnu.org/ml/gcc/2017-06/msg00044.html https://gcc.gnu.org/ml/gcc/2017-06/msg00044.html https://lists.debian.org/debian-devel/2017/06/msg00433.html https://lists.debian.org/debian-devel/2017/06/msg00433.html It was a bit bizarre when he hit the Octave mailing list. Eventually, people just wanted xz back: http://octave.1599824.n4.nabble.com/opinion-bring-back-Octave-xz-source-release-td4683705.html http://octave.1599824.n4.nabble.com/opinion-bring-back-Octav...

AdmiralAsshat 7y ago
Everyone's gotta have their white whale, I guess.

esaym 7y ago

Interestingly, since "recovery" is mentioned several times, I decided to test myself. I took a copy of a jpeg image, compressed it different times with either gzip or bzip2, then with a hexeditor modified one byte. The recovery instructions for gzip is to simply do "zcat corrupt_file.gz > corrupt_file". While for bzip2 is to use the bzip2recover command which just dumps the blocks out individually (corrupt ones and all). Uncompressing the corrupt gzip jpeg file via zcat at all times resulted in an image file the same size as the original and could be opened with any image viewer although the colors were clearly off. I never could recover the image compressed with bzip2. Trying to extract all the recovered blocks made by bzip2recover via bzcat would just choke on the single corrupted block. And the smallest you can make a block is 100K (vs 32K for gzip?). Obviously pulling 100K out of a jpeg will not work. Though I'm still confused as to how the corrupted gzip file extracted to a file of the same size as the original. I guess gzip writes out the corrupted data as well instead of choking on it? I guess gzip is the winner here. Having a file with a corrupted byte is much better than having a file with 100K of data missing...

wereHamster 7y ago
Whether recovery leads to (almost) useable data depends on what byte you modify. It's entirely possible that a single corrupt byte in the compressed data leads to a single corrupt byte when uncompressed. When you are dealing with images you may not even notice that a single pixel is wrong. But it's also possible that you completely destroy the data such that the decompression algorithm can't even deal with it and has to give up.
- chasil 7y ago
  A decade and a half ago, I wrote an Oracle archived log that I had compressed with bzip2 to a DLT40 tape. I recovered and uncompressed (without error) the log, then tried to apply it to a database recovery which rejected it as corrupt. After several attempts to read the tape (amounting to dozens of hours), I finally put it in the original drive that wrote it and pulled the file to the remote recovery system - this worked. I immediately began including PAR2 files on the tapes, so the restored contents could be verified and corrected. I have my doubts that bzip2 is as sensitive to corruption as the author of asserts, but perhaps there have been improvements to the code since my misfortune.

asveikau 7y ago

I remember seeing this article before. This time the reaction that surges for me is: if you want long-term archiving but don't assume redundant storage, it's not going to go well. Put your long-term archives on ZFS.

StavrosK 7y ago
Why are you assuming they aren't assuming redundant storage? Redundant storage isn't a cure-all, there's still a chance two blocks on two disks will fail in the exact same spot.
- kipari 7y ago
  I reckon that the chance of the same two blocks on two different disks failing between ZFS scrubs would be incredibly small.
  - StavrosK 7y ago
    Yes, over normal timescales. A lot can happen in a thousand years.
    - asveikau 7y ago
      Thousands of years is a lot of scrubs and a lot of disk replacements, though. And a solution like ZFS, properly monitored, should help make those detections and repairs happen early, with lower odds of loss. Although honestly in a thousand year timeframe I very much doubt humanity will preserve ZFS, gzip, tar, jpeg, PNG, ASCII, today's spoken and written languages in current form, etc. Just as written material from 1000 years ago is not very accessible to most people; with the original material you need intense study before you even know what you're looking at.
  - cmurf 7y ago
    It is incredibly small if you don't consider either drive failing. But if one drive fails, it happens with some regularity that a sector on the good drive is bad. In actuality, only one sector is bad, but in effect the dead drive means its mirror is also bad. This comes up on the linux raid list with some frequency whenever there are drive failures with raid56, and the subsequently the raid trips over a single bad sector. But it's true that lack of scrubbing contributes to this scenario, as well as the terrible combination of consumer drives with very high bad sector recovery times and the Linux SCSI command timer default of 30 seconds. That combination ends up causing a masking of bad sectors that end up not getting repaired, and as a user you may not realize that the link resets are not normal and suggest a bad sector as the cause.

xoa 7y ago

Not that many of the complaints aren't reasonable, but I thought that in general compression/format was orthogonal to parity, which is what I assume is actually wanted for long-term archiving? I always figured that the goal should normally to be able to get back out a bit-perfect copy of whatever went in, using something like Parchive at the file level or ZFS for online storage at the fs level. I guess on the principle of layers and graceful failure modes it's better if even sub-archives can handle some level of corruption without total failure, and from a long term perspective of implementation independence simpler/better specified is preferable, but that still doesn't seem to substitute for just having enough parity built in to both notice corruption and fully recover from it to fairly extreme levels.

matt-attack 7y ago
I think with archiving it’s more than that. Sure you can guarantee that the actual tool you just compressed with can restore the original perfectly. But with long term digital archiving I think you need the assurance that the “spec” called “xz” could be perfectly reimplemented by an expert in the future. Based solely on documentation. And on a platform that doesn’t exist today. That is, you must assume the original executable is either not available or not able to be executed.
- xoa 7y ago
  I don't disagree with that at all! I did try to say (maybe unclearly) that having simpler more foolproof and failsafe layers at every level seems absolutely worth pursuing anyway, where possible. But I also wonder whether some of the common wisdom is from an age that is obsolete for many scenarios? Ie., in the 80s and 90s and even early 00s there was a lot more churn, practices were less standardized, computing time was more expensive, storage capacity was far more expensive, the ratio of software to data size was higher, etc. The latter seems to tie into "must assume the original executable is either not available or not able to be executed." For serious archiving, does it no make sense to just bundle in not merely the executable, but in some instances an entire environment? In a "fall of civilization" type scenario that may not be helpful vs a clear simple spec and ease of bootstrapping, but for situations where technological continuity is a limiting factor for some reason anyway is it safe to simply assume basics like at this point x86 will never go away as something that is at least virtualized? In my own experience there is a pretty clear cutoff date after which I can continue to run the entire environment in a VM. Again this is shooting the breeze a bit, article is discussing a case where there should be the freedom to choose better formats. But for a lot of important archive material, including software itself, are we getting to the point where many long term archives should simply including everything necessary to deal with them in the present day as a container or VM image, which is then stored with a solid amount of parity and replication?

shmerl 7y ago

xz-utils should implement parallel decompression already. pixz is doing it, but stock xz is not. Most end users benefit from faster decompression.

ltbarcly3 7y ago

No file format is perfect, I've been using xz for years and I can't think of a single issue I have had. The compression rate is dramatically better than gzip or bzip2 for many types of archives (especially when there is a large redundancy, for example when compressing spidered web pages from the same site you can get well over 99% size reduction compared to 70% reduction for gzip, which means using less than one 30th of the disk space). Lately I have been using zstd for some things since it gives good compression and is much faster than xz. This criticism of xz just seems nit picky and impractical, especially if you are compressing tar archives and/or storing the archives on some kind of raid which can correct some read errors (such as raid5).

mkj 7y ago

A bit of speculation here, but perhaps xz won over lzip because it has a real manpage? lzip has the usual infuriating short summary of options with a "run info lzip for the complete manual". Also the source code repository doesn't even seem linked directly from the lzip homepage - technical considerations aren't the only thing that determines if software is "better", it also has to be well presented.