12 ms·

Tar is an ill-specified format (2015)

112 points by moomin 6y ago

ceratin6 6y ago
Fuck you to whoever thinks it’s ok to diss on tar. Your mom was ill-specified.
- dang 6y ago
  I don't think this comment should have been flagged. It's obviously not literal. However, could you please stop creating accounts for every few comments you post? We ban accounts that do that. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix&page=0&query=community%20identity%20by:dang https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
  - Wowfunhappy 6y ago
    > I don't think this comment should have been flagged. It's obviously not literal. ...it’s not at all obvious to me...
    - dang 6y ago
      It's absurd to get that worked up about tar.
yowlingcat 6y ago
Really? And all this time, I was hoping that it wouldn't end up being a tarpit.
- ceratin6 6y ago
  It’s all these modern smokers with their low-tar cigarettes. Tar is tar. Let it be. tar -czvf if you’re gzzy. tar+gzip beats zip for most uses. I don’t know about whether 7z beats tar+gz, but I don’t think 7z is as ubiquitous, though that may not be true now.
  - QuercusMax 6y ago
    7zip always seemed really shady, almost intentionally so compared to zip and tar+gz. Always gave me the "leet haxor" vibe. 7zip makes me feel like I need to wash my hands.
    - GekkePrutser 6y ago
      7zip was a really old format for compatibility with 7-bit data streams. I used it all the time with packet radio in the early 90s. A bit like zipping and then base64 encoding does. But I don't know how it got to the wares/hacker scene from there. I assume they will also have dropped the 7 bit thing as it would reduce efficiency by 1/8th on modern systems.
      - Kwpolska 6y ago
        That's likely a different 7zip. The 7z format was invented in 1999, and the Wikipedia page [0] doesn't say anything about 7-bit compatibility. [0] https://en.wikipedia.org/wiki/7z https://en.wikipedia.org/wiki/7z
      - lstamour 6y ago
        Edit: as another commenter rightly points out, it's likely unrelated. Here's the changelog: https://www.7-zip.org/history.txt https://www.7-zip.org/history.txt That said the first entry at the bottom dates back to 2.00, so it's possible there was a 1.0 that predates this changelog: 2.00 Beta 1 1999-01-02 - Original beta version. I'm pretty sure its popularity increased because it was one of the few options on Windows that was completely free to install with no nag screens, ever. It had a decent right-click menu; excellent usability for its time. And you wouldn't be tempted to use an older copy such as whatever shipped with Windows XP to extract archives. Plus it offered compatibility with all other formats including its own, and eventually apps like Firefox publicly shipped with 7zip Self-Extractors and such. And if you wanted to test whether 7zip really was more efficient, you could quickly try compressing files using different options - in my personal testing, 7zip often won back then. Not sure if results have changed recently given there are a few new compression techniques these days.
chungy 6y ago
Archive link: https://web.archive.org/web/20210329212251/https://invisible-island.net/autoconf/portability-tar.html https://web.archive.org/web/20210329212251/https://invisible...
LeoPanthera 6y ago
This isn't wrong, but it's mildly misleading. Pax is mentioned right at the top, but glosses over that pax creates "ustar" format tar files, which is specified by POSIX.1-2001. ustar tar files are compatible with almost all tar archivers, and are in fact well-specified.
tdeck 6y ago
The other common archive, ZIP, has a standard. But it's a hot mess! To quote Wikipedia: > Tools that correctly read ZIP archives must scan for the end of central directory record signature, and then, as appropriate, the other, indicated, central directory records. They must not scan for entries from the top of the ZIP file, because (as previously mentioned in this section) only the central directory specifies where a file chunk starts and that it has not been deleted. Scanning could lead to false positives, as the format does not forbid other data to be between chunks, nor file data streams from containing such signatures You might think that ZIP files would have a header, or at least a fixed-width footer, but they don't. Instead you're supposed to scan backward through the file looking for a magic number indicating you've found the central directory. It's amazing to me that this is the format that won the compression wars and stuck around to this day.
- LeoPanthera 6y ago
  zip at least has a de-facto standard, which today is the "Info-ZIP" utility.
  - chungy 6y ago
    Zip also has a real bonefide standard: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
    - greggman3 6y ago
      That spec is seriously underspecified https://games.greggman.com/game/zip-rant/ https://games.greggman.com/game/zip-rant/
      - chungy 6y ago
        Seems kind of a shame to tuck this away in a comments section on a (mostly) unrelated article. I would suggest you submit it as a main article for HN :)
  - 6y ago
mixologic 6y ago
In my 28+ years of working with tar files I have ran into issues exactly zero times with being able to tar or untar something. Im sure if you're deeper down the stack this ends up being one of those nuanced peas under a mattress, but Im guessing that 99% of people who use tar do not run into issues using tar.
- dehrmann 6y ago
  I recently discovered that you can't append to a tar file quickly. It caused me a lot of headaches with incremental updates. http://tiamat.name/blogposts/fast-appending-files-to-tar-archive-is-impossible/ http://tiamat.name/blogposts/fast-appending-files-to-tar-arc...
  - st_goliath 6y ago
    > You’d think that -r option usage forces tar application to append files to the end of the archive, getting the position of the archive’s end from archive’s index. It doesn’t. Tar format is designed in a way that it has no index. Um, yes. A tar file is just a blob of 512 byte aligned files with 512 byte headers in front of them + an optional 1k null-bytes signifying the end. Technically all there is to appending at the end is to slap on another header and a file blob? EDIT: Ok, on second thought, you do need a linear scan. There could be ancillary data after the 1k termination blob. Even if not an intentional polyglot nonsense, but if you actually use tar on a main frame tape drive as originally intended, simply seeking to the end doesn't work.
    - gumby 6y ago
      Yes because the name ‘tar’ means ‘Tape ARchive’. Appending to the end is how you would add a file to a tape. In fact tar will silently overwrite an existing file because that is how you would update a file on a tape (that functionality is arguably useful today for other reasons)
- fanf2 6y ago
  I’ve been using various flavours of unix for a similar length of time. I always installed gnu tar and gzip, and then I had no issues.
st_goliath 6y ago
I once foolishly thought, I'll write a tar parser because, "how hard can it be" [1]. I simply tried to follow the tar(5) man page[2], and got a reference test set from another website posted previously on HN[3]. Along the way I discovered that NetBSD pax apparently cannot handle the PAX format[3] and my parser inadvertently uncovered that git-archive was storing the checksums wrong, but nobody noticed because other tar implementations were more lax about it[4]. As the article describes (as does the man page), tar is actually a really simple format, but there are just so many variants to choose from. Turns out, if you strive for maximum compatibility, it's easiest to stick to what GNU tar does and favor GNU extensions over PAX header fields. If you think about it, in many ways the GNU project IMO ended up doing "embrace, extend, extinguish" with Unix. [1] https://github.com/AgentD/squashfs-tools-ng/tree/master/lib/tar https://github.com/AgentD/squashfs-tools-ng/tree/master/lib/... [2] https://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5 https://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5 [3] https://mgorny.pl/articles/portability-of-tar-features.html https://mgorny.pl/articles/portability-of-tar-features.html [4] https://www.spinics.net/lists/git/msg363049.html https://www.spinics.net/lists/git/msg363049.html
- slaymaker1907 6y ago
  There are similar problems with csv. A deceptively simple format with many variants to handle various edge cases.
  - lstamour 6y ago
    Yes, except it's easy to debug a csv problem because it's generally text in some encoding you can hopefully identify without looking at specific byte offsets in a hex editor: http://fileformats.archiveteam.org/wiki/Tape_Archive#Identification http://fileformats.archiveteam.org/wiki/Tape_Archive#Identif... In fact, one of the biggest questions I've still yet to answer is whether Docker or OCI have ever clarified just which TAR format must be followed? As far as I can tell, it's a mix of "whatever library X supports"? https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for example. I'm not sure I want a more complicated format, but I do wish somebody would pick a new file extension for their next use of tar so we could clearly differentiate one tar format from another the next time someone builds on top of simple tar files for their binary distribution, etc.
mongol 6y ago
Sqlite has an archive format that is worthwhile to be aware of https://sqlite.org/sqlar.html https://sqlite.org/sqlar.html
- slaymaker1907 6y ago
  It unfortunately has major limitations on file size for use cases where that matters. Max file size is on the order of 2^32 bytes since files are stored in a single SQLite blob. Note that this limit is only for input file size. The archive file can basically be of any arbitrary size. Personally I wish sqlar didn't have this limitation since SQLite is awesome for tooling.
  - dragonwriter 6y ago
    It seems pretty easy to build big files variant of sqlar; instead of a single sqlar table, have index and filedata tables; the index has all the elements of the normal sqlar table except data, and adds a counter for the number of chunks the file is stored in (a directory is zero chunks, an empty file is zero-length with one, empty, chunk), the filedata has composite PK consisting of an FK to the index and an integer sequence number in the file, and a data blob. The code changes to store/extract from it should be straightforward, and given the size of SQLite integers it would in theory structurally handle file sizes (as stored, which may be compressed) of up to 2^63-1 bytes; but the SQLite 281 TB database size limit would be reached well before this.
gumby 6y ago
Random trivia: gnu tar was called gnu tar before it became the base of the GNU project’s tar. Why? Because John Gilmore’s username has been gnu since the early 80s. I still call him gnu as he of course calls me gumby.
- person_of_color 6y ago
  Just how rich did Sun make John Gilmore?
brongondwana 6y ago
Yeah, maybe... but I wrote https://metacpan.org/pod/Archive::Tar::Stream https://metacpan.org/pod/Archive::Tar::Stream without too much trouble and it's been happily managing all our backups for over 10 years since (it didn't get released to CPAN straight away). It helps that we're only ever appending gzipped tar fragments, and we have a tracking database format to know where the offsets in the uncompressed stream are: https://fastmail.blog/2015/12/21/building-a-backup-system-for-cyrus/ https://fastmail.blog/2015/12/21/building-a-backup-system-fo... But yeah - much worse if you're parsing arbitrary tar files than if you create everything yourself and just have to make it readable by gnu tar and that's about it.
qwerty456127 6y ago
Tar should have been abandoned a decade ago, in favour of 7z or something like that. It baffles me it's 2021 an we still have to extract an entire archive just to look at the list of files it contains, let alone get a particular file. I've never heard a valid reason to use tar on today servers (let alone PCs) other than "it's guaranteed to be available everywhere".
- cestith 6y ago
  > we still have to extract an entire archive just to look at the list of files GNU tar has -t and --list, while NetBSD, OpenBSD, and FreeBSD all have -t. None of them extract the archive to list the contents as far as I am aware.
  - fl0wenol 6y ago
    I think they mean that you have to read in the whole archive (incl. decompression) to see all the files; that there's no option for an index that can be immediately seeked.
    - cestith 6y ago
      Since tar is a streaming format originally intended for tape and compression is done via external programs against the full archive, it's a little difficult to look inside the archive if it's compressed. Other formats accomplish this by compressing individual archive members inline with building the archive then building a TOC that's not compressed. That's all implementation details. As a user, I can use tar and an outside compression tool together in one command to see the file list. tar -Jtf foo.tar.bzip tar -xtf example.tar.gz tar -I /usr/bin/mydecompressor -tf somethingelse.tar.myc So unless there's some huge, huge archive out there that takes forever to decompress and read it's not really an issue for the user.
      - qwerty456127 6y ago
        > Since tar is a streaming format originally intended for tape So why are we still using it if ~99% of Linux users and administrators have never seen a digital tape drive? I've built quite a number of Linux, BSD and Windows servers yet the only tape with digital data I have seen in my entire life was for a Spectrum. Do we really need our files stored in a way compatible with tape storage? > As a user, I can use tar and an outside compression tool together in one command to see the file list. Which would decompress the whole archive, the seek through it. Ridiculous waste with no benefit. Tar is the ultimate idiosyncrasy of of Linux.
bestouff 6y ago
I also have written my own tar parser, in https://github.com/bestouff/genext2fs https://github.com/bestouff/genext2fs - oh boy it's not easy ! My parser is of course incomplete and won't handle corner cases, which are really plenty. In the end I added libarchive as an alternative to my hand-rolled code because that's how it works best.
gorgoiler 6y ago
It’s a shame you can’t cat tar archives together. It always seemed a neat feature of container formats like mpeg: cat chapter*.mpg >movie.mpg Are there any file archive formats that support this?
- hddqsb 6y ago
  For some reason I thought this was a standard feature of tar, and was surprised when it didn't work. It turns out that GNU tar supports extraction of concatenated tarballs, but it requires the --ignore-zeros (-i) option: > Normally, tar stops reading when it encounters a block of zeros between file entries (which usually indicates the end of the archive). ‘--ignore-zeros’ (‘-i’) allows tar to completely read an archive which contains a block of zeros before the end (i.e., a damaged archive, or one that was created by concatenating several archives together). -- https://www.gnu.org/software/tar/manual/html_node/Ignore-Zeros.html https://www.gnu.org/software/tar/manual/html_node/Ignore-Zer... (As a bonus, this also works for concatenated .tar.gz files because gzip supports concatenation.) Regarding other formats that support concatenation, the Linux kernel's initramfs format is interesting because it is based on CPIO but is explicitly defined to be the concatenation of CPIO archives: https://www.kernel.org/doc/html/latest/driver-api/early-userspace/buffer-format.html https://www.kernel.org/doc/html/latest/driver-api/early-user... CPIO itself doesn't support concatenation. There is a neat hack to extract such concatenated archives: while cpio -i; do :; done < archive.cpio (source: https://unix.stackexchange.com/a/266090; https://unix.stackexchange.com/a/266090; also mentioned there is the tool skipcpio, which is part of dracut: https://github.com/dracutdevs/dracut/blob/master/skipcpio/skipcpio.c https://github.com/dracutdevs/dracut/blob/master/skipcpio/sk...) This hack would also work for extracting concatenating tarballs (without GNU tar's --ignore-zeros option). One annoyance is that it shows a warning when it gets to the end of the archive.
- LeoPanthera 6y ago
  You can cat .mpg files together, but you should not. ".mpg" is supposed to refer to an MPEG Program Stream, which does in fact have a header. You can cat Transport Streams, but these are supposed to have the ".ts" extension. You sometimes get transports streams named ".mpg", which complicates everything.
geokon 6y ago
... is there an accepted better alternative?
xurukefi 6y ago
Whilst we're at it: does anybody know what's the deal with the weird octal ASCII representation in the tar header? To store a file size of 10 bytes, all common tar implementations typically put literally the string "0000012" into the header, i.e., the size in octal in ASCII representation. I stumbled upon this when I had to write a parser because I wanted to rename files inside a tar without extracting the files first (turns out there is no tool for that). I found this format quite peculiar because I don't see any benefit of doing it this way.
- st_goliath 6y ago
  Tar was in Unix very early on. The first versions of Unix were IIRC written for a PDP-8 and later ported to a PDP-11. Most of the PDP era DEC machines are designed favoring an octal encoding. The PDP-8 was a 12 bit machine (12=4 * 3). They also had few 18 (=6 * 3) and 36 bit (=12 * 3) machines. The PDP-11 being a 16 bit machine with 8 bit granular memory access was an odd man out, but still had many aspects designed around splitting words into groups of 3 bits, e.g. if you look at the instruction set encoding. Or just take a look at the front panel[1]. Because of the DEC machines that early Unix was developed on (and the people involved in early Unix development being very familiar with them), octal encoding crept into many places where it stayed until today, including the C programing language and some Unix specific file formats. Using octal, stored as plain text ASCII makes it very easy for a human to debug, for whom reading octal is second hand nature. [1] https://en.wikipedia.org/wiki/File:Pdp-11-70-panel.jpg https://en.wikipedia.org/wiki/File:Pdp-11-70-panel.jpg
  - xurukefi 6y ago
    Very interesting, thx!
thomashabets2 6y ago
cpio is pretty shitty too. After reading the manpage I found a pretty serious bug: https://nvd.nist.gov/vuln/detail/CVE-2019-14866 https://nvd.nist.gov/vuln/detail/CVE-2019-14866 https://blog.habets.se/2019/11/CVE-2019-14866-gnu-cpio.html https://blog.habets.se/2019/11/CVE-2019-14866-gnu-cpio.html

112 points by moomin 6y ago

ceratin6 6y ago

Fuck you to whoever thinks it’s ok to diss on tar. Your mom was ill-specified.

dang 6y ago
I don't think this comment should have been flagged. It's obviously not literal. However, could you please stop creating accounts for every few comments you post? We ban accounts that do that. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix&page=0&query=community%20identity%20by:dang https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
- Wowfunhappy 6y ago
  > I don't think this comment should have been flagged. It's obviously not literal. ...it’s not at all obvious to me...
  - dang 6y ago
    It's absurd to get that worked up about tar.

yowlingcat 6y ago

Really? And all this time, I was hoping that it wouldn't end up being a tarpit.

ceratin6 6y ago
It’s all these modern smokers with their low-tar cigarettes. Tar is tar. Let it be. tar -czvf if you’re gzzy. tar+gzip beats zip for most uses. I don’t know about whether 7z beats tar+gz, but I don’t think 7z is as ubiquitous, though that may not be true now.
- QuercusMax 6y ago
  7zip always seemed really shady, almost intentionally so compared to zip and tar+gz. Always gave me the "leet haxor" vibe. 7zip makes me feel like I need to wash my hands.
  - GekkePrutser 6y ago
    7zip was a really old format for compatibility with 7-bit data streams. I used it all the time with packet radio in the early 90s. A bit like zipping and then base64 encoding does. But I don't know how it got to the wares/hacker scene from there. I assume they will also have dropped the 7 bit thing as it would reduce efficiency by 1/8th on modern systems.
    - Kwpolska 6y ago
      That's likely a different 7zip. The 7z format was invented in 1999, and the Wikipedia page [0] doesn't say anything about 7-bit compatibility. [0] https://en.wikipedia.org/wiki/7z https://en.wikipedia.org/wiki/7z
    - lstamour 6y ago
      Edit: as another commenter rightly points out, it's likely unrelated. Here's the changelog: https://www.7-zip.org/history.txt https://www.7-zip.org/history.txt That said the first entry at the bottom dates back to 2.00, so it's possible there was a 1.0 that predates this changelog: 2.00 Beta 1 1999-01-02 - Original beta version. I'm pretty sure its popularity increased because it was one of the few options on Windows that was completely free to install with no nag screens, ever. It had a decent right-click menu; excellent usability for its time. And you wouldn't be tempted to use an older copy such as whatever shipped with Windows XP to extract archives. Plus it offered compatibility with all other formats including its own, and eventually apps like Firefox publicly shipped with 7zip Self-Extractors and such. And if you wanted to test whether 7zip really was more efficient, you could quickly try compressing files using different options - in my personal testing, 7zip often won back then. Not sure if results have changed recently given there are a few new compression techniques these days.

chungy 6y ago

Archive link: https://web.archive.org/web/20210329212251/https://invisible-island.net/autoconf/portability-tar.html https://web.archive.org/web/20210329212251/https://invisible...

LeoPanthera 6y ago

This isn't wrong, but it's mildly misleading. Pax is mentioned right at the top, but glosses over that pax creates "ustar" format tar files, which is specified by POSIX.1-2001. ustar tar files are compatible with almost all tar archivers, and are in fact well-specified.

tdeck 6y ago

The other common archive, ZIP, has a standard. But it's a hot mess! To quote Wikipedia: > Tools that correctly read ZIP archives must scan for the end of central directory record signature, and then, as appropriate, the other, indicated, central directory records. They must not scan for entries from the top of the ZIP file, because (as previously mentioned in this section) only the central directory specifies where a file chunk starts and that it has not been deleted. Scanning could lead to false positives, as the format does not forbid other data to be between chunks, nor file data streams from containing such signatures You might think that ZIP files would have a header, or at least a fixed-width footer, but they don't. Instead you're supposed to scan backward through the file looking for a magic number indicating you've found the central directory. It's amazing to me that this is the format that won the compression wars and stuck around to this day.

LeoPanthera 6y ago
zip at least has a de-facto standard, which today is the "Info-ZIP" utility.
- chungy 6y ago
  Zip also has a real bonefide standard: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
  - greggman3 6y ago
    That spec is seriously underspecified https://games.greggman.com/game/zip-rant/ https://games.greggman.com/game/zip-rant/
    - chungy 6y ago
      Seems kind of a shame to tuck this away in a comments section on a (mostly) unrelated article. I would suggest you submit it as a main article for HN :)
- 6y ago

mixologic 6y ago

In my 28+ years of working with tar files I have ran into issues exactly zero times with being able to tar or untar something. Im sure if you're deeper down the stack this ends up being one of those nuanced peas under a mattress, but Im guessing that 99% of people who use tar do not run into issues using tar.

dehrmann 6y ago
I recently discovered that you can't append to a tar file quickly. It caused me a lot of headaches with incremental updates. http://tiamat.name/blogposts/fast-appending-files-to-tar-archive-is-impossible/ http://tiamat.name/blogposts/fast-appending-files-to-tar-arc...
- st_goliath 6y ago
  > You’d think that -r option usage forces tar application to append files to the end of the archive, getting the position of the archive’s end from archive’s index. It doesn’t. Tar format is designed in a way that it has no index. Um, yes. A tar file is just a blob of 512 byte aligned files with 512 byte headers in front of them + an optional 1k null-bytes signifying the end. Technically all there is to appending at the end is to slap on another header and a file blob? EDIT: Ok, on second thought, you do need a linear scan. There could be ancillary data after the 1k termination blob. Even if not an intentional polyglot nonsense, but if you actually use tar on a main frame tape drive as originally intended, simply seeking to the end doesn't work.
  - gumby 6y ago
    Yes because the name ‘tar’ means ‘Tape ARchive’. Appending to the end is how you would add a file to a tape. In fact tar will silently overwrite an existing file because that is how you would update a file on a tape (that functionality is arguably useful today for other reasons)
fanf2 6y ago
I’ve been using various flavours of unix for a similar length of time. I always installed gnu tar and gzip, and then I had no issues.

st_goliath 6y ago

I once foolishly thought, I'll write a tar parser because, "how hard can it be" [1]. I simply tried to follow the tar(5) man page[2], and got a reference test set from another website posted previously on HN[3]. Along the way I discovered that NetBSD pax apparently cannot handle the PAX format[3] and my parser inadvertently uncovered that git-archive was storing the checksums wrong, but nobody noticed because other tar implementations were more lax about it[4]. As the article describes (as does the man page), tar is actually a really simple format, but there are just so many variants to choose from. Turns out, if you strive for maximum compatibility, it's easiest to stick to what GNU tar does and favor GNU extensions over PAX header fields. If you think about it, in many ways the GNU project IMO ended up doing "embrace, extend, extinguish" with Unix. [1] https://github.com/AgentD/squashfs-tools-ng/tree/master/lib/tar https://github.com/AgentD/squashfs-tools-ng/tree/master/lib/... [2] https://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5 https://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5 [3] https://mgorny.pl/articles/portability-of-tar-features.html https://mgorny.pl/articles/portability-of-tar-features.html [4] https://www.spinics.net/lists/git/msg363049.html https://www.spinics.net/lists/git/msg363049.html

slaymaker1907 6y ago
There are similar problems with csv. A deceptively simple format with many variants to handle various edge cases.
- lstamour 6y ago
  Yes, except it's easy to debug a csv problem because it's generally text in some encoding you can hopefully identify without looking at specific byte offsets in a hex editor: http://fileformats.archiveteam.org/wiki/Tape_Archive#Identification http://fileformats.archiveteam.org/wiki/Tape_Archive#Identif... In fact, one of the biggest questions I've still yet to answer is whether Docker or OCI have ever clarified just which TAR format must be followed? As far as I can tell, it's a mix of "whatever library X supports"? https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for example. I'm not sure I want a more complicated format, but I do wish somebody would pick a new file extension for their next use of tar so we could clearly differentiate one tar format from another the next time someone builds on top of simple tar files for their binary distribution, etc.

mongol 6y ago

Sqlite has an archive format that is worthwhile to be aware of https://sqlite.org/sqlar.html https://sqlite.org/sqlar.html

slaymaker1907 6y ago
It unfortunately has major limitations on file size for use cases where that matters. Max file size is on the order of 2^32 bytes since files are stored in a single SQLite blob. Note that this limit is only for input file size. The archive file can basically be of any arbitrary size. Personally I wish sqlar didn't have this limitation since SQLite is awesome for tooling.
- dragonwriter 6y ago
  It seems pretty easy to build big files variant of sqlar; instead of a single sqlar table, have index and filedata tables; the index has all the elements of the normal sqlar table except data, and adds a counter for the number of chunks the file is stored in (a directory is zero chunks, an empty file is zero-length with one, empty, chunk), the filedata has composite PK consisting of an FK to the index and an integer sequence number in the file, and a data blob. The code changes to store/extract from it should be straightforward, and given the size of SQLite integers it would in theory structurally handle file sizes (as stored, which may be compressed) of up to 2^63-1 bytes; but the SQLite 281 TB database size limit would be reached well before this.

gumby 6y ago

Random trivia: gnu tar was called gnu tar before it became the base of the GNU project’s tar. Why? Because John Gilmore’s username has been gnu since the early 80s. I still call him gnu as he of course calls me gumby.

person_of_color 6y ago
Just how rich did Sun make John Gilmore?

brongondwana 6y ago

Yeah, maybe... but I wrote https://metacpan.org/pod/Archive::Tar::Stream https://metacpan.org/pod/Archive::Tar::Stream without too much trouble and it's been happily managing all our backups for over 10 years since (it didn't get released to CPAN straight away). It helps that we're only ever appending gzipped tar fragments, and we have a tracking database format to know where the offsets in the uncompressed stream are: https://fastmail.blog/2015/12/21/building-a-backup-system-for-cyrus/ https://fastmail.blog/2015/12/21/building-a-backup-system-fo... But yeah - much worse if you're parsing arbitrary tar files than if you create everything yourself and just have to make it readable by gnu tar and that's about it.

qwerty456127 6y ago

Tar should have been abandoned a decade ago, in favour of 7z or something like that. It baffles me it's 2021 an we still have to extract an entire archive just to look at the list of files it contains, let alone get a particular file. I've never heard a valid reason to use tar on today servers (let alone PCs) other than "it's guaranteed to be available everywhere".

cestith 6y ago
> we still have to extract an entire archive just to look at the list of files GNU tar has -t and --list, while NetBSD, OpenBSD, and FreeBSD all have -t. None of them extract the archive to list the contents as far as I am aware.
- fl0wenol 6y ago
  I think they mean that you have to read in the whole archive (incl. decompression) to see all the files; that there's no option for an index that can be immediately seeked.
  - cestith 6y ago
    Since tar is a streaming format originally intended for tape and compression is done via external programs against the full archive, it's a little difficult to look inside the archive if it's compressed. Other formats accomplish this by compressing individual archive members inline with building the archive then building a TOC that's not compressed. That's all implementation details. As a user, I can use tar and an outside compression tool together in one command to see the file list. tar -Jtf foo.tar.bzip tar -xtf example.tar.gz tar -I /usr/bin/mydecompressor -tf somethingelse.tar.myc So unless there's some huge, huge archive out there that takes forever to decompress and read it's not really an issue for the user.
    - qwerty456127 6y ago
      > Since tar is a streaming format originally intended for tape So why are we still using it if ~99% of Linux users and administrators have never seen a digital tape drive? I've built quite a number of Linux, BSD and Windows servers yet the only tape with digital data I have seen in my entire life was for a Spectrum. Do we really need our files stored in a way compatible with tape storage? > As a user, I can use tar and an outside compression tool together in one command to see the file list. Which would decompress the whole archive, the seek through it. Ridiculous waste with no benefit. Tar is the ultimate idiosyncrasy of of Linux.

bestouff 6y ago

I also have written my own tar parser, in https://github.com/bestouff/genext2fs https://github.com/bestouff/genext2fs - oh boy it's not easy ! My parser is of course incomplete and won't handle corner cases, which are really plenty. In the end I added libarchive as an alternative to my hand-rolled code because that's how it works best.

gorgoiler 6y ago

It’s a shame you can’t cat tar archives together. It always seemed a neat feature of container formats like mpeg: cat chapter*.mpg >movie.mpg Are there any file archive formats that support this?

hddqsb 6y ago
For some reason I thought this was a standard feature of tar, and was surprised when it didn't work. It turns out that GNU tar supports extraction of concatenated tarballs, but it requires the --ignore-zeros (-i) option: > Normally, tar stops reading when it encounters a block of zeros between file entries (which usually indicates the end of the archive). ‘--ignore-zeros’ (‘-i’) allows tar to completely read an archive which contains a block of zeros before the end (i.e., a damaged archive, or one that was created by concatenating several archives together). -- https://www.gnu.org/software/tar/manual/html_node/Ignore-Zeros.html https://www.gnu.org/software/tar/manual/html_node/Ignore-Zer... (As a bonus, this also works for concatenated .tar.gz files because gzip supports concatenation.) Regarding other formats that support concatenation, the Linux kernel's initramfs format is interesting because it is based on CPIO but is explicitly defined to be the concatenation of CPIO archives: https://www.kernel.org/doc/html/latest/driver-api/early-userspace/buffer-format.html https://www.kernel.org/doc/html/latest/driver-api/early-user... CPIO itself doesn't support concatenation. There is a neat hack to extract such concatenated archives: while cpio -i; do :; done < archive.cpio (source: https://unix.stackexchange.com/a/266090; https://unix.stackexchange.com/a/266090; also mentioned there is the tool skipcpio, which is part of dracut: https://github.com/dracutdevs/dracut/blob/master/skipcpio/skipcpio.c https://github.com/dracutdevs/dracut/blob/master/skipcpio/sk...) This hack would also work for extracting concatenating tarballs (without GNU tar's --ignore-zeros option). One annoyance is that it shows a warning when it gets to the end of the archive.
LeoPanthera 6y ago
You can cat .mpg files together, but you should not. ".mpg" is supposed to refer to an MPEG Program Stream, which does in fact have a header. You can cat Transport Streams, but these are supposed to have the ".ts" extension. You sometimes get transports streams named ".mpg", which complicates everything.

geokon 6y ago

... is there an accepted better alternative?

xurukefi 6y ago

Whilst we're at it: does anybody know what's the deal with the weird octal ASCII representation in the tar header? To store a file size of 10 bytes, all common tar implementations typically put literally the string "0000012" into the header, i.e., the size in octal in ASCII representation. I stumbled upon this when I had to write a parser because I wanted to rename files inside a tar without extracting the files first (turns out there is no tool for that). I found this format quite peculiar because I don't see any benefit of doing it this way.

st_goliath 6y ago
Tar was in Unix very early on. The first versions of Unix were IIRC written for a PDP-8 and later ported to a PDP-11. Most of the PDP era DEC machines are designed favoring an octal encoding. The PDP-8 was a 12 bit machine (12=4 * 3). They also had few 18 (=6 * 3) and 36 bit (=12 * 3) machines. The PDP-11 being a 16 bit machine with 8 bit granular memory access was an odd man out, but still had many aspects designed around splitting words into groups of 3 bits, e.g. if you look at the instruction set encoding. Or just take a look at the front panel[1]. Because of the DEC machines that early Unix was developed on (and the people involved in early Unix development being very familiar with them), octal encoding crept into many places where it stayed until today, including the C programing language and some Unix specific file formats. Using octal, stored as plain text ASCII makes it very easy for a human to debug, for whom reading octal is second hand nature. [1] https://en.wikipedia.org/wiki/File:Pdp-11-70-panel.jpg https://en.wikipedia.org/wiki/File:Pdp-11-70-panel.jpg
- xurukefi 6y ago
  Very interesting, thx!

thomashabets2 6y ago

cpio is pretty shitty too. After reading the manpage I found a pretty serious bug: https://nvd.nist.gov/vuln/detail/CVE-2019-14866 https://nvd.nist.gov/vuln/detail/CVE-2019-14866 https://blog.habets.se/2019/11/CVE-2019-14866-gnu-cpio.html https://blog.habets.se/2019/11/CVE-2019-14866-gnu-cpio.html