33 ms·

Please add (I believe this should be trivial) all the tar's metadata-related features (I mean saving links and access rights). There should be only one, all the

Please add (I believe this should be trivial) all the tar's metadata-related features (I mean saving links and access rights). There should be only one, all the archive formats different from 7z have to die. TAR and ZIP are both horrible. and should be displaced by 7Z.

c0l0 4y ago
This is kinda naive to wish for - tar has applications that reach beyond stuffing a BLOB containing other BLOBs and some metadata into a seekable file somewhere, such as archiving data to magnetic tape/LTO. And even if you develop the perfect archive format that happens to nail every possible use-case 100% (which you won't, because there just are too many), you will STILL have to deal with compressed and archived artifacts which accumulated over the last six or so decades in various places.
- qwerty456127 4y ago
  > tar has applications... magnetic tape/LTO. That's a parallel universe far far away, ~99% of people having to use tar (just because it's the standard and the only format supporting links/access metadata) every day will never have to use a tape drive. I don't see a reason why we can't use 2 separate formats - one for everyday packaging, a different one for tape drives. > And even if you develop the perfect archive format There is a perfect format already - it's 7z. Just add file access rights and links support to it. No need to invent anything really new.
  - kelnos 4y ago
    > There is a perfect format already - it's 7z. Just add file access rights and links support to it. These two sentences seem to contradict each other.
    - qwerty456127 4y ago
      Not really. Adding support for a particular kind of metadata is a change so minuscule it doesn't qualify as a change in the format. Apple just stores their filesystem metadata as in a special sub-directory in ZIP files. And the only problem with the Apple's solution is nobody else respects it. 7zip is a format developed and maintained by a specific author who is alive and active so he can just build the same in the standard 7zip implementations and chances are everybody will accept. By the way I have just found an actual imperfection in 7zip: it can't let you choose the order in which archived files are stored in it nor chose different compression parameters for specific files. This limits its applicability. E.g. the EPUB standard says the first file in an archive must be "mimetype" and it must be not compressed. But I believe this can be fixed with reasonable ease (and probably without breaking changes) as well.
chriswarbo 4y ago
7z makes the same mistake as zip: implementing both a filesystem and a compression algorithm. There's no way a single tool can implement all the bells and whistles of the various filesystems in use today. For example, skimming through the 7zip format ( https://py7zr.readthedocs.io/en/latest/archive_format.html https://py7zr.readthedocs.io/en/latest/archive_format.html ) I can see specific support for file attributes from (current versions of) Windows and Unix, but no AmigaOS protect bits ( http://www.jaruzel.com/amiga/amiga-os-command-reference-help/protect.html http://www.jaruzel.com/amiga/amiga-os-command-reference-help... ) Keeping these two tasks separate allows swapping-out the implementation of each (e.g. I tend to use .tar.lz these days, since I'm mostly on Unix)
- qwerty456127 4y ago
  I understand your point yet the whole point of mine is straight opposite.
- afiori 4y ago
  once you have a general file-system representation format (with all its complexities) adding compression of the blobs seems a minor addition. On a related note I was surprised to discover that the Windows 10 backup tool was able to store about 240GB of various data in barely 80GB of backup; I believe it must have spliced most files to look for common fragment to deduplicate (with some NTFS magic behind maybe). .tar.xz will never be able to do that, if I try to compress 10 copies of the entire firefox codebase, it will never be able to recognize the duplicated files; only something like a file-system+compression could do that.
  - 5e92cb50239222b 4y ago
    > only something like a file-system+compression could do that borg handles this just fine. I put all kinds of stuff into borg repositories: raw MySQL/PostgreSQL data directories, tar archives (both compressed and uncompressed), or just / recursively. You can do stuff like: $ tar -caf - / | borg create … or even $ borg create … </dev/sda and your repository grows by the amount of data changed since last backup (or by a couple of kilobytes if nothing has changed). https://github.com/borgbackup/borg https://github.com/borgbackup/borg

c0l0 4y ago

This is kinda naive to wish for - tar has applications that reach beyond stuffing a BLOB containing other BLOBs and some metadata into a seekable file somewhere, such as archiving data to magnetic tape/LTO. And even if you develop the perfect archive format that happens to nail every possible use-case 100% (which you won't, because there just are too many), you will STILL have to deal with compressed and archived artifacts which accumulated over the last six or so decades in various places.

qwerty456127 4y ago
> tar has applications... magnetic tape/LTO. That's a parallel universe far far away, ~99% of people having to use tar (just because it's the standard and the only format supporting links/access metadata) every day will never have to use a tape drive. I don't see a reason why we can't use 2 separate formats - one for everyday packaging, a different one for tape drives. > And even if you develop the perfect archive format There is a perfect format already - it's 7z. Just add file access rights and links support to it. No need to invent anything really new.
- kelnos 4y ago
  > There is a perfect format already - it's 7z. Just add file access rights and links support to it. These two sentences seem to contradict each other.
  - qwerty456127 4y ago
    Not really. Adding support for a particular kind of metadata is a change so minuscule it doesn't qualify as a change in the format. Apple just stores their filesystem metadata as in a special sub-directory in ZIP files. And the only problem with the Apple's solution is nobody else respects it. 7zip is a format developed and maintained by a specific author who is alive and active so he can just build the same in the standard 7zip implementations and chances are everybody will accept. By the way I have just found an actual imperfection in 7zip: it can't let you choose the order in which archived files are stored in it nor chose different compression parameters for specific files. This limits its applicability. E.g. the EPUB standard says the first file in an archive must be "mimetype" and it must be not compressed. But I believe this can be fixed with reasonable ease (and probably without breaking changes) as well.

chriswarbo 4y ago

7z makes the same mistake as zip: implementing both a filesystem and a compression algorithm. There's no way a single tool can implement all the bells and whistles of the various filesystems in use today. For example, skimming through the 7zip format ( https://py7zr.readthedocs.io/en/latest/archive_format.html https://py7zr.readthedocs.io/en/latest/archive_format.html ) I can see specific support for file attributes from (current versions of) Windows and Unix, but no AmigaOS protect bits ( http://www.jaruzel.com/amiga/amiga-os-command-reference-help/protect.html http://www.jaruzel.com/amiga/amiga-os-command-reference-help... ) Keeping these two tasks separate allows swapping-out the implementation of each (e.g. I tend to use .tar.lz these days, since I'm mostly on Unix)

qwerty456127 4y ago
I understand your point yet the whole point of mine is straight opposite.
afiori 4y ago
once you have a general file-system representation format (with all its complexities) adding compression of the blobs seems a minor addition. On a related note I was surprised to discover that the Windows 10 backup tool was able to store about 240GB of various data in barely 80GB of backup; I believe it must have spliced most files to look for common fragment to deduplicate (with some NTFS magic behind maybe). .tar.xz will never be able to do that, if I try to compress 10 copies of the entire firefox codebase, it will never be able to recognize the duplicated files; only something like a file-system+compression could do that.
- 5e92cb50239222b 4y ago
  > only something like a file-system+compression could do that borg handles this just fine. I put all kinds of stuff into borg repositories: raw MySQL/PostgreSQL data directories, tar archives (both compressed and uncompressed), or just / recursively. You can do stuff like: $ tar -caf - / | borg create … or even $ borg create … </dev/sda and your repository grows by the amount of data changed since last backup (or by a couple of kilobytes if nothing has changed). https://github.com/borgbackup/borg https://github.com/borgbackup/borg