4 ms·

If recovery is critical, use lzip and not xz. The compression ratio is the same but xz archives are particularly vulnerable to corruption.

mangamadaiyan 5y ago
Out of curiosity, what in the file format makes it particularly vulnerable to corruption?
- mangamadaiyan 5y ago
  A little googling threw up this link: https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html I haven't read through the link properly, so I'm as yet unable to comment on the merits of xz or the lack thereof; posting it here for other readers who might be curious, as I was.
  - walrus01 5y ago
    I can't say that I've ever seen a corrupt xz file that wasn't in a place where a bzip2 or lzip or anything else wouldn't have also been hosed (a very thoroughly failed ext4fs file system on a dying hard drive).
    - bombcar 5y ago
      It could be a situation where it's "how bad are you hosed" - for example a bit flipping in a text document can pretty easily be fixed by hand (though it could be completely missed, and critical to be sure) - but a compressed format could either recover losing a "sector" or so, or simply be impossible to restore anything from it. There's an argument that you're better off knowing you lost data than having most but with a hidden flaw - but either way you want multiple copies and preferably in a format that contains error detection AND correction - and itself is stored on a filesystem that provides similar (ZFS for example).
adrianN 5y ago
If recovery is critical, store some parity bits too. par2 or something like that.
squarefoot 5y ago
I wouldn't use compression at all wherever possible. Those old drives are likely at least two orders of magnitude smaller than modern ones, so storing everything uncompressed, or even the entire disk image shouldn't be problematic. Plain uncompressed files would also be easier to recover in the unlikely, but possible, event of failure of the media used to salvage them to.
dmos62 5y ago
I think hard drive images are rarely compressed, because it's just so slow.
Chris2048 5y ago
Also, it can be a nice trick to take a fs-structure dump: list of all files on disk, with associated sizes, permissions, and md5 hash. It should be much smaller (and faster) to take such a snapshot, but priceless if you ever suffer corruption for e.g identifying & verifying inodes, restoring permissions etc. Also useful for identifying which files are vanilla (same as installed from package) so can be restored by other means (finding original package). For that reason, a dump of installed packages w/ version is also useful.

mangamadaiyan 5y ago

Out of curiosity, what in the file format makes it particularly vulnerable to corruption?

mangamadaiyan 5y ago
A little googling threw up this link: https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html I haven't read through the link properly, so I'm as yet unable to comment on the merits of xz or the lack thereof; posting it here for other readers who might be curious, as I was.
- walrus01 5y ago
  I can't say that I've ever seen a corrupt xz file that wasn't in a place where a bzip2 or lzip or anything else wouldn't have also been hosed (a very thoroughly failed ext4fs file system on a dying hard drive).
  - bombcar 5y ago
    It could be a situation where it's "how bad are you hosed" - for example a bit flipping in a text document can pretty easily be fixed by hand (though it could be completely missed, and critical to be sure) - but a compressed format could either recover losing a "sector" or so, or simply be impossible to restore anything from it. There's an argument that you're better off knowing you lost data than having most but with a hidden flaw - but either way you want multiple copies and preferably in a format that contains error detection AND correction - and itself is stored on a filesystem that provides similar (ZFS for example).

adrianN 5y ago

If recovery is critical, store some parity bits too. par2 or something like that.

squarefoot 5y ago

I wouldn't use compression at all wherever possible. Those old drives are likely at least two orders of magnitude smaller than modern ones, so storing everything uncompressed, or even the entire disk image shouldn't be problematic. Plain uncompressed files would also be easier to recover in the unlikely, but possible, event of failure of the media used to salvage them to.

dmos62 5y ago

I think hard drive images are rarely compressed, because it's just so slow.

Chris2048 5y ago

Also, it can be a nice trick to take a fs-structure dump: list of all files on disk, with associated sizes, permissions, and md5 hash. It should be much smaller (and faster) to take such a snapshot, but priceless if you ever suffer corruption for e.g identifying & verifying inodes, restoring permissions etc. Also useful for identifying which files are vanilla (same as installed from package) so can be restored by other means (finding original package). For that reason, a dump of installed packages w/ version is also useful.