3 ms·

A bit of shameless self-promotion, I wrote a blog post some time ago which goes through the history of tar[1]. Tar definitely has evolved through the Unix wars,

by cyphar 6y ago

A bit of shameless self-promotion, I wrote a blog post some time ago which goes through the history of tar[1]. Tar definitely has evolved through the Unix wars, but I wouldn't say the final result is really that great. While we do tend to reinvent tools for reinvention's sake, I do think tar really does deserve a redesign based on modern constraints (rather than the constraints of tape archives). Zip archives are markedly better designed. Also, cpio did evolve at least somewhat. In fact POSIX has a separate archive tool called "pax" (which supports both tar and cpio) which was meant to act as a truce between the tar and cpio factions of Unix vendors. Nobody uses it, as far as I know. [1]: https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar

lboc 6y ago
It sees some use in z/OS. For example, open source tarballs aren't generally EBCDIC, so: pax -rvf automake-1.16.tar -o from=ISO8859-1,to=IBM-1047
masklinn 6y ago
> Zip archives are markedly better designed. Yes and no. They have advantages and drawbacks: * zip archives don't require seeking through the entire thing to list archive contents * zip archives can random-access files cheaply * zip checksums the files themselves (AFAIK tar only checksums the header) * zip compresses files individually, meaning different files can use different configuration * the central directory (CD) means you can trade efficiency for very cheap updates: removing a file is just rewriting the CD without that file, likewise to replace a file just tack the new file at the end, then write a new CD with updated offsets However * "streaming" zips can lead to schizophrenia as the central directory record does not necessarily match the streamed contents or ordering * zip only compresses files individually meaning there's no cross-file efficiency to be had (some compression formats have a "solid" archive concept which compresses globally as an opt-in) so tar can compressed better depending on the mix and the compression algorithm (assuming both archives use the same method) > Also, cpio did evolve at least somewhat. In fact POSIX has a separate archive tool called "pax" (which supports both tar and cpio) which was meant to act as a truce between the tar and cpio factions of Unix vendors. Nobody uses it, as far as I know. GNU CPIO apparently straights support tar.
- cyphar 6y ago
  I would argue the upsides out-weigh the downsides. Out of two downsides you mention, one actually applies to tar archives as well (the ordering of files can be random) -- it's just that there's no index to compare against. If you added an index to a tar archive it would have the same issue. The compression point you make is fair, though there is a clear trade-off when it comes to random-access. Naive whole-archive compression makes efficient random-accesses impossible because you have to decompress everything before the data you're trying to access. A sufficiently-clever archive and compression format may be able to overcome that, but I'd argue that zip made a reasonable trade-off here. > AFAIK tar only checksums the header tar doesn't checksum anything, there isn't even a CRC.
  - masklinn 6y ago
    > I would argue the upsides out-weigh the downsides. I could agree, but that doesn't mean replacing tars by zips doesn't have drawbacks. Even more so as you can use whatever compression scheme you want with tars, not so with zips (unless you use them as tars and then… they're just a worse tar). > Out of two downsides you mention, one actually applies to tar archives as well (the ordering of files can be random) -- it's just that there's no index to compare against. There is no schyzophrenia because tars always behave that way. > If you added an index to a tar archive it would have the same issue. Yes, if tar were a completely different format it would have different issues. It's not though. > Naive whole-archive compression makes efficient random-accesses impossible because you have to decompress everything before the data you're trying to access. A sufficiently-clever archive and compression format may be able to overcome that, but I'd argue that zip made a reasonable trade-off here. So did tar, they're just different tradeoffs. For instance archiving tons of small files with zip is an absolutely miserable experience because you've got more overhead than you have content for instance. Because tar was designed for tape archive, its use-cases included backups which, for university computer, would be likely to include lots of small files. > tar doesn't checksum anything, there isn't even a CRC. It's not a CRC, but tar has a very simple checksum on headers: the 257 bytes of the header are summed into an integer.

by cyphar 6y ago

lboc 6y ago

It sees some use in z/OS. For example, open source tarballs aren't generally EBCDIC, so: pax -rvf automake-1.16.tar -o from=ISO8859-1,to=IBM-1047

masklinn 6y ago

> Zip archives are markedly better designed. Yes and no. They have advantages and drawbacks: * zip archives don't require seeking through the entire thing to list archive contents * zip archives can random-access files cheaply * zip checksums the files themselves (AFAIK tar only checksums the header) * zip compresses files individually, meaning different files can use different configuration * the central directory (CD) means you can trade efficiency for very cheap updates: removing a file is just rewriting the CD without that file, likewise to replace a file just tack the new file at the end, then write a new CD with updated offsets However * "streaming" zips can lead to schizophrenia as the central directory record does not necessarily match the streamed contents or ordering * zip only compresses files individually meaning there's no cross-file efficiency to be had (some compression formats have a "solid" archive concept which compresses globally as an opt-in) so tar can compressed better depending on the mix and the compression algorithm (assuming both archives use the same method) > Also, cpio did evolve at least somewhat. In fact POSIX has a separate archive tool called "pax" (which supports both tar and cpio) which was meant to act as a truce between the tar and cpio factions of Unix vendors. Nobody uses it, as far as I know. GNU CPIO apparently straights support tar.

cyphar 6y ago
I would argue the upsides out-weigh the downsides. Out of two downsides you mention, one actually applies to tar archives as well (the ordering of files can be random) -- it's just that there's no index to compare against. If you added an index to a tar archive it would have the same issue. The compression point you make is fair, though there is a clear trade-off when it comes to random-access. Naive whole-archive compression makes efficient random-accesses impossible because you have to decompress everything before the data you're trying to access. A sufficiently-clever archive and compression format may be able to overcome that, but I'd argue that zip made a reasonable trade-off here. > AFAIK tar only checksums the header tar doesn't checksum anything, there isn't even a CRC.
- masklinn 6y ago
  > I would argue the upsides out-weigh the downsides. I could agree, but that doesn't mean replacing tars by zips doesn't have drawbacks. Even more so as you can use whatever compression scheme you want with tars, not so with zips (unless you use them as tars and then… they're just a worse tar). > Out of two downsides you mention, one actually applies to tar archives as well (the ordering of files can be random) -- it's just that there's no index to compare against. There is no schyzophrenia because tars always behave that way. > If you added an index to a tar archive it would have the same issue. Yes, if tar were a completely different format it would have different issues. It's not though. > Naive whole-archive compression makes efficient random-accesses impossible because you have to decompress everything before the data you're trying to access. A sufficiently-clever archive and compression format may be able to overcome that, but I'd argue that zip made a reasonable trade-off here. So did tar, they're just different tradeoffs. For instance archiving tons of small files with zip is an absolutely miserable experience because you've got more overhead than you have content for instance. Because tar was designed for tape archive, its use-cases included backups which, for university computer, would be likely to include lots of small files. > tar doesn't checksum anything, there isn't even a CRC. It's not a CRC, but tar has a very simple checksum on headers: the 257 bytes of the header are summed into an integer.