3 ms·

Folks seem to be comparing xz to zstd, but if I am understanding correctly the true competitor to xz is the article author’s “lzip” format, which uses the same

Folks seem to be comparing xz to zstd, but if I am understanding correctly the true competitor to xz is the article author’s “lzip” format, which uses the same LZMA compression as xz but with a much better designed container format (at least according to the author).

exyi 4y ago
I'd not say it's necessarily better designed - it's just simpler. Few bytes of some headers, LZMA compressed data, checksum, done. No support for seeks and stuff like that
loeg 4y ago
Emphasis on “according to the author,” yeah. And the concerns are silly — FEC can and should be added outside the compressed data (e.g., with par2).
- brnt 4y ago
  But why outside the compression container? I'd love to have format that let me repair their contents up to the number of parity bits it stored.
  - joveian 4y ago
    I agree that a single file is super convenient. It looks like the in-progress par3 will support that (par3 inside zip, I'd add an mtree file as well for metadata if needed): https://parchive.github.io/doc/Parity_Volume_Set_Specification_v3.0.html https://parchive.github.io/doc/Parity_Volume_Set_Specificati... I don't use par2 right now for backups due to the extra hastle, although I might use par3. It might end up a waste of space, though, since I haven't had an issue with corrupted files in at least 20 years (would be great for optical media that can get scratched). For the longest term archiving I'm not sure any kind of compression is a good idea, I'd think along the lines of uncompressed copy + parity info in a simple container where all files are contiguous (maybe a raw zip file without compression).
    - brnt 4y ago
      The par3 spec has been in development for so long that I have a hard head in it's completion and uptake. I've thought of other integrations as well. A file system that would have integrated parity (much like ZFS/btrfs checksumming, but then recoverable up to the parity you allocated). Of even integrated into 'regular' file formats: there could be an MP3 tag/header that holds parity for it's data stream. Alas, it seems that people have decided this will be solved through other means, par2 is rare as it is. Shameless plug: I wrote a GUI tool to make it slightly easier to work with par2 and a big dir of archive data: https://pypi.org/project/par2deep/ https://pypi.org/project/par2deep/

exyi 4y ago

I'd not say it's necessarily better designed - it's just simpler. Few bytes of some headers, LZMA compressed data, checksum, done. No support for seeks and stuff like that

loeg 4y ago

Emphasis on “according to the author,” yeah. And the concerns are silly — FEC can and should be added outside the compressed data (e.g., with par2).

brnt 4y ago
But why outside the compression container? I'd love to have format that let me repair their contents up to the number of parity bits it stored.
- joveian 4y ago
  I agree that a single file is super convenient. It looks like the in-progress par3 will support that (par3 inside zip, I'd add an mtree file as well for metadata if needed): https://parchive.github.io/doc/Parity_Volume_Set_Specification_v3.0.html https://parchive.github.io/doc/Parity_Volume_Set_Specificati... I don't use par2 right now for backups due to the extra hastle, although I might use par3. It might end up a waste of space, though, since I haven't had an issue with corrupted files in at least 20 years (would be great for optical media that can get scratched). For the longest term archiving I'm not sure any kind of compression is a good idea, I'd think along the lines of uncompressed copy + parity info in a simple container where all files are contiguous (maybe a raw zip file without compression).
  - brnt 4y ago
    The par3 spec has been in development for so long that I have a hard head in it's completion and uptake. I've thought of other integrations as well. A file system that would have integrated parity (much like ZFS/btrfs checksumming, but then recoverable up to the parity you allocated). Of even integrated into 'regular' file formats: there could be an MP3 tag/header that holds parity for it's data stream. Alas, it seems that people have decided this will be solved through other means, par2 is rare as it is. Shameless plug: I wrote a GUI tool to make it slightly easier to work with par2 and a big dir of archive data: https://pypi.org/project/par2deep/ https://pypi.org/project/par2deep/