5 ms·

I have no idea why you think that use case does not exist. Your whole idea about archive seems to be it is to ensure a blob doesn't change (same hash). But that

I have no idea why you think that use case does not exist. Your whole idea about archive seems to be it is to ensure a blob doesn't change (same hash). But that's far from the only use of archive. (Hell, even with that, you are assuming you know the correct hash of the file to begin with, which isn't guaranteed.) "Repairing" corrupt archives, as in to get as much as usable data from that archive is a pretty useful thing and I have done it multiple times. For example, an archive can have hundreds of files inside and if you can recover any of them that's better than nothing. It is also one of the reason I still use WinRAR occasionally due to its great recovery record (RR) feature. >replaced with a fresh copy from replicated storage Lots of times you don't have other copy.

jmillikin 4y ago
The process of long-term archival starts with replication. A common approach is two local copies on separate physical media, and one remote copy in a cloud storage service with add-only permissions. This protects against hardware failure, bad software (accidental deletion, malware), natural disasters (flood, fire) and other 99th-percentile disaster conditions. The cloud storage providers will have their own level of replication (AWS S3 has a 99.999999999% durability SLA). If you have only one copy of some important file and you discover it no longer matches the stored checksum, then that's not a question of archival, but of data recovery. There's no plausible mechanism by which a file might suffer a few bitflips and be saved by careful application of a CRC -- bitrot often zeros out entire sectors.
- Sakos 4y ago
  It's simply not sufficient to have multiple copies. It's way too easy to propagate errors in a way that slips under the radar, which then screws you over 5 years later. The main idea of long-time archival is redundancy. Replication is one form of redundancy, but it's not the only one and not the only one you should use.
  - jmillikin 4y ago
    This is nonsense. Replication to storage in different failure domains is quite sufficient to ensure long-term data preservation, and errors cannot "propagate" to archives unless your risk model involves angry wizards.
    - grosswait 4y ago
      Bit rot is a thing. It seems like you have a different idea of what archival means than most of us.
- exmadscientist 4y ago
  > There's no plausible mechanism by which a file might suffer a few bitflips and be saved by careful application of a CRC -- bitrot often zeros out entire sectors. A CRC, no, absolutely not. But this is exactly what PAR2 recovery records do, they do it well, and they (or their equivalents) need to be easier to enable in more places. Setting up a replication and durability scheme is a major pain in the ass. Passing the `--add-recovery-record` switch on the command line is very, very easy, and it is good enough for many cases where "best effort" protection against corruption is all that is needed.
- mike_hock 4y ago
  None of this addresses the criticism levied in the article, nor does it defend xz's inconsistent design decisions that are all over the place. Why shouldn't we try to squeeze the highest rate of data recovery out of the unlikely event that we're left with the only remaining copy if it costs nothing extra (just the choice of one archiver over another)? Should you choose xz for future archival purposes? No. Should Debian make an active effort to switch away from xz now? Probably not, as their primary concern is distribution, not archival, and xz is good enough.
  - jmillikin 4y ago
    > None of this addresses the criticism levied in the article The article's criticism isn't worth addressing (or reading). Nothing it complains about is important. > nor does it defend xz's inconsistent design decisions that are all > over the place. The level of inconsistency that the article complains about doesn't matter. Pretty much every popular format looks like that. Try writing a Matroska or PDF decoder some time. > Why shouldn't we try to squeeze the highest rate of data recovery > out of the unlikely event that we're left with the only remaining > copy if it costs nothing extra Because it isn't important. For cases where recovery of data from a corrupt compressed stream is important, you'd wrap the compressed data in a container with built-in error correction. Then you'd use that format's error correction to recover the correct compressed data stream, and feed that to your decompressor. > Should you choose xz for future archival purposes? No. Yes you should. XZ is fine, despite the article's silliness. It's better than gzip or bzip2 and nearly as widely supported. If there's a replacement for XZ as a general-purpose compression format for archived data then it'll be selected on the quality of its compression, not whether the bitstream format can produce valid output from invalid input.
    - mike_hock 4y ago
      > The level of inconsistency that the article complains about doesn't matter. Pretty much every popular format looks like that. Try writing a Matroska or PDF decoder some time. So a format should be designed wrong because other formats are also designed wrong. > Because it isn't important. So? You don't have to go out of your way to micro-optimize, others like TFA's author are already doing it for you. You just have to pick the micro-optimized product off the shelf. "It isn't important" is a complete non-argument when the bad choices are made for no reason and no benefit at all and especially not even making the design process quicker and easier. > If there's a replacement for XZ as a general-purpose compression format for archived data then it'll be selected on the quality of its compression, not whether the bitstream format can produce valid output from invalid input. The article also addresses design problems that harm the compression ratio.
maxloh 4y ago
In 2022, it should be quite rare that an archive got corrupted. Internet connection is so much better now and almost 100% of downloads completed successfully.
- usr1106 4y ago
  To my own surprise I have seen a corrupted file in 2022. Only 70 MB, so really tiny compared to many files handled today. The file had been built in Europe and after installing it to a US system we wondered why it did not work. Haven't seen anything like that for many years so it took us a while to even consider the option that it could be corrupted. (No, we did not spend any time to check whether corruption showed any interesting pattern like a single bit flip, a block of zeros or anything like that. Transferring it again just did it.)
- pmoriarty 4y ago
  The larger the file the more chance it will get corrupted by cosmic rays. Media also physically decays over time. Error correction and redundancy is essential, which is why I use par2 and dvdisaster on all my archives.
  - maxloh 4y ago
    Shouldn't the download client check for corruption before any download complete?

jmillikin 4y ago

The process of long-term archival starts with replication. A common approach is two local copies on separate physical media, and one remote copy in a cloud storage service with add-only permissions. This protects against hardware failure, bad software (accidental deletion, malware), natural disasters (flood, fire) and other 99th-percentile disaster conditions. The cloud storage providers will have their own level of replication (AWS S3 has a 99.999999999% durability SLA). If you have only one copy of some important file and you discover it no longer matches the stored checksum, then that's not a question of archival, but of data recovery. There's no plausible mechanism by which a file might suffer a few bitflips and be saved by careful application of a CRC -- bitrot often zeros out entire sectors.

Sakos 4y ago
It's simply not sufficient to have multiple copies. It's way too easy to propagate errors in a way that slips under the radar, which then screws you over 5 years later. The main idea of long-time archival is redundancy. Replication is one form of redundancy, but it's not the only one and not the only one you should use.
- jmillikin 4y ago
  This is nonsense. Replication to storage in different failure domains is quite sufficient to ensure long-term data preservation, and errors cannot "propagate" to archives unless your risk model involves angry wizards.
  - grosswait 4y ago
    Bit rot is a thing. It seems like you have a different idea of what archival means than most of us.
exmadscientist 4y ago
> There's no plausible mechanism by which a file might suffer a few bitflips and be saved by careful application of a CRC -- bitrot often zeros out entire sectors. A CRC, no, absolutely not. But this is exactly what PAR2 recovery records do, they do it well, and they (or their equivalents) need to be easier to enable in more places. Setting up a replication and durability scheme is a major pain in the ass. Passing the `--add-recovery-record` switch on the command line is very, very easy, and it is good enough for many cases where "best effort" protection against corruption is all that is needed.
mike_hock 4y ago
None of this addresses the criticism levied in the article, nor does it defend xz's inconsistent design decisions that are all over the place. Why shouldn't we try to squeeze the highest rate of data recovery out of the unlikely event that we're left with the only remaining copy if it costs nothing extra (just the choice of one archiver over another)? Should you choose xz for future archival purposes? No. Should Debian make an active effort to switch away from xz now? Probably not, as their primary concern is distribution, not archival, and xz is good enough.
- jmillikin 4y ago
  > None of this addresses the criticism levied in the article The article's criticism isn't worth addressing (or reading). Nothing it complains about is important. > nor does it defend xz's inconsistent design decisions that are all > over the place. The level of inconsistency that the article complains about doesn't matter. Pretty much every popular format looks like that. Try writing a Matroska or PDF decoder some time. > Why shouldn't we try to squeeze the highest rate of data recovery > out of the unlikely event that we're left with the only remaining > copy if it costs nothing extra Because it isn't important. For cases where recovery of data from a corrupt compressed stream is important, you'd wrap the compressed data in a container with built-in error correction. Then you'd use that format's error correction to recover the correct compressed data stream, and feed that to your decompressor. > Should you choose xz for future archival purposes? No. Yes you should. XZ is fine, despite the article's silliness. It's better than gzip or bzip2 and nearly as widely supported. If there's a replacement for XZ as a general-purpose compression format for archived data then it'll be selected on the quality of its compression, not whether the bitstream format can produce valid output from invalid input.
  - mike_hock 4y ago
    > The level of inconsistency that the article complains about doesn't matter. Pretty much every popular format looks like that. Try writing a Matroska or PDF decoder some time. So a format should be designed wrong because other formats are also designed wrong. > Because it isn't important. So? You don't have to go out of your way to micro-optimize, others like TFA's author are already doing it for you. You just have to pick the micro-optimized product off the shelf. "It isn't important" is a complete non-argument when the bad choices are made for no reason and no benefit at all and especially not even making the design process quicker and easier. > If there's a replacement for XZ as a general-purpose compression format for archived data then it'll be selected on the quality of its compression, not whether the bitstream format can produce valid output from invalid input. The article also addresses design problems that harm the compression ratio.

maxloh 4y ago

In 2022, it should be quite rare that an archive got corrupted. Internet connection is so much better now and almost 100% of downloads completed successfully.

usr1106 4y ago
To my own surprise I have seen a corrupted file in 2022. Only 70 MB, so really tiny compared to many files handled today. The file had been built in Europe and after installing it to a US system we wondered why it did not work. Haven't seen anything like that for many years so it took us a while to even consider the option that it could be corrupted. (No, we did not spend any time to check whether corruption showed any interesting pattern like a single bit flip, a block of zeros or anything like that. Transferring it again just did it.)
pmoriarty 4y ago
The larger the file the more chance it will get corrupted by cosmic rays. Media also physically decays over time. Error correction and redundancy is essential, which is why I use par2 and dvdisaster on all my archives.
- maxloh 4y ago
  Shouldn't the download client check for corruption before any download complete?