4 ms·
Coincidentally, I recently submitted an article by Thomas E. Dickey on the subject of tar and portability. * https://news.ycombinator.com/item?id=18969290 http
by JdeBP 8y ago
Coincidentally, I recently submitted an article by Thomas E. Dickey on the subject of tar and portability.
* https://news.ycombinator.com/item?id=18969290 https://news.ycombinator.com/item?id=18969290
The hoo-ha over tar and cpio in the 1980s when the subject of standardization came up, and the whole raison d'être of pax, is largely forgotten nowadays. But the problems that were brought to light back then live on.
The thing to bear in mind is that we already have attempts to improve upon this. The problematic formats in the 1980s were themselves products of the second-system effect, adding on various things to the original tape archive format. We are now, in the second decade of the 21st century, well into umpteenth-system effect.
This brings up a point where this article is very wrong. The pax utility did not appear in 2001. pax was in the POSIX draft back in the early 1990s, and had become widespread enough to be in reference books by 1991. The article conflates the PAX extensions with the pax utility. This, and the incorrect dates, are errors promulgated by Wikipedia, which I challenged back in 2018 at https://en.wikipedia.org/wiki/Talk:Pax_(Unix)#Incomplete_Information https://en.wikipedia.org/wiki/Talk:Pax_(Unix)#Incomplete_Inf... and which was probably the source of these errors in the article. Always double-check what Wikipedia says on computing topics with a proper reference.
Before being tempted to reinvent the wheel here, I recommend a look at all of the times during the so-called "archiver wars" where this wheel already was reinvented, and learning from them. Of particular note, given this article, is Rahul Dhesi's ZOO file format from 1986. It allowed for multiple generations of a given file, and archive headers marked deleted files with a flag (allowing them to be undeleted), which could be used for "whiteouts". It suffers far less from an extension mess because support for filesystem features from Unix, MS-DOS, and VMS (at least as they were in 1986) was provided in the base data strutures; including 8.3 and long filenames.
But really the basic error here is in using an off-line archive format for an on-line live filesystem mechanism. It's the wrong data structure for the job. Whereas there are right data structures, and have been for years. The history of filesystem formats development includes several cases of addressing the very things mentioned in the article, from deduplication (c.f. ZFS) through generations for deleted files (ODS for Files-11 in VMS, where ZOO got the idea from) to reproducible directory scan orders (c.f. the side-effects of using B-trees in HPFS).
- cyphar 8y ago> This, and the incorrect dates, are errors promulgated by Wikipedia [...] which was probably the source of these errors in the article. Always double-check what Wikipedia says on computing topics with a proper reference. I didn't use Wikipedia, actually. The primary source was the libarchive documentation, star, bits of GNU tar's docs, and POSIX. The main issue is that it's hard to get a copy of POSIX.1-1988, let alone POSIX drafts from the 1990s. EDIT: Also, I didn't actually notice there was a rationale section in POSIX.1-2001 which references PAX as existing in earlier standards. I will read through it and update my article accordingly. Thank you! > Of particular note, given this article, is Rahul Dhesi's ZOO file format from 1986. Funnily enough, I have heard of ZOO (not sure where) and looked into it. While it does support file versions (and deletion) and has many improvements over tar, there are many other properties it doesn't have that we'd need (and last I checked there's no real support for it in modern Linux and Unix-likes -- so it makes no difference from a ubiquity perspective). > But really the basic error here is in using an off-line archive format for an on-line live filesystem mechanism. We don't use tar archives as the live filesystem for containers, it's used as a distribution mechanism. > from deduplication (c.f. ZFS) While ZFS has de-duplication, it's not really the kind we need and (from memory) zfs-send doesn't include the de-dup tables so they're all generated on the receiving end. Ideally we'd want content-defined de-duplication because that way you can reproducibly generate the blobs.
- JdeBP 8y agoI suggest books. (-: Fred Zlotnick's 1991 book mentions pax and the (contemporary) POSIX.2 draft. Mark Horton's Portable C Software from 1990 has a command options listing for pax. That's just two of the books.