5 ms·

In my 28+ years of working with tar files I have ran into issues exactly zero times with being able to tar or untar something. Im sure if you're deeper down th

In my 28+ years of working with tar files I have ran into issues exactly zero times with being able to tar or untar something. Im sure if you're deeper down the stack this ends up being one of those nuanced peas under a mattress, but Im guessing that 99% of people who use tar do not run into issues using tar.

dehrmann 6y ago
I recently discovered that you can't append to a tar file quickly. It caused me a lot of headaches with incremental updates. http://tiamat.name/blogposts/fast-appending-files-to-tar-archive-is-impossible/ http://tiamat.name/blogposts/fast-appending-files-to-tar-arc...
- st_goliath 6y ago
  > You’d think that -r option usage forces tar application to append files to the end of the archive, getting the position of the archive’s end from archive’s index. It doesn’t. Tar format is designed in a way that it has no index. Um, yes. A tar file is just a blob of 512 byte aligned files with 512 byte headers in front of them + an optional 1k null-bytes signifying the end. Technically all there is to appending at the end is to slap on another header and a file blob? EDIT: Ok, on second thought, you do need a linear scan. There could be ancillary data after the 1k termination blob. Even if not an intentional polyglot nonsense, but if you actually use tar on a main frame tape drive as originally intended, simply seeking to the end doesn't work.
  - gumby 6y ago
    Yes because the name ‘tar’ means ‘Tape ARchive’. Appending to the end is how you would add a file to a tape. In fact tar will silently overwrite an existing file because that is how you would update a file on a tape (that functionality is arguably useful today for other reasons)
fanf2 6y ago
I’ve been using various flavours of unix for a similar length of time. I always installed gnu tar and gzip, and then I had no issues.
jopsen 6y ago
Have you tried rolling a custom tar implementation? That's when you realize how many cases you actually have to handle.
- josefx 6y ago
  Did a quick and dirty implementation to read a file from Java. Immediately ran into the "ustar " vs. "ustar" magic strings used to differentiate between format versions as I generated files on several different systems. That lead to some head scratching because I couldn't comprehend why there would be a difference on that last whitespace.
regularfry 6y ago
So, there is a direct effect of this that's really annoying when you bump into it. Docker images are tarballs, and the layer cache relies on how the tar file is constructed. The same layer content constructed by two different tar implementations will appear different to the cache lookup. For reasons known only to themselves, Docker originally used different implementations for the docker command-line tool and the docker-compose tool. This means that cache entries produced by two tools that you got from the same place don't match up, and you can't use one tool to cache a layer and have the other pick it up. At least, not by default - if you know about it, there's a switch you can flip to have one use the other's implementation. But they had to add that in, and do that extra work, because tar isn't well-specified.
- rozab 6y ago
  Seems like gzip would be the bigger issue there. There are millions of valid ways to compress a single file

dehrmann 6y ago

I recently discovered that you can't append to a tar file quickly. It caused me a lot of headaches with incremental updates. http://tiamat.name/blogposts/fast-appending-files-to-tar-archive-is-impossible/ http://tiamat.name/blogposts/fast-appending-files-to-tar-arc...

st_goliath 6y ago
> You’d think that -r option usage forces tar application to append files to the end of the archive, getting the position of the archive’s end from archive’s index. It doesn’t. Tar format is designed in a way that it has no index. Um, yes. A tar file is just a blob of 512 byte aligned files with 512 byte headers in front of them + an optional 1k null-bytes signifying the end. Technically all there is to appending at the end is to slap on another header and a file blob? EDIT: Ok, on second thought, you do need a linear scan. There could be ancillary data after the 1k termination blob. Even if not an intentional polyglot nonsense, but if you actually use tar on a main frame tape drive as originally intended, simply seeking to the end doesn't work.
- gumby 6y ago
  Yes because the name ‘tar’ means ‘Tape ARchive’. Appending to the end is how you would add a file to a tape. In fact tar will silently overwrite an existing file because that is how you would update a file on a tape (that functionality is arguably useful today for other reasons)

fanf2 6y ago

I’ve been using various flavours of unix for a similar length of time. I always installed gnu tar and gzip, and then I had no issues.

jopsen 6y ago

Have you tried rolling a custom tar implementation? That's when you realize how many cases you actually have to handle.

josefx 6y ago
Did a quick and dirty implementation to read a file from Java. Immediately ran into the "ustar " vs. "ustar" magic strings used to differentiate between format versions as I generated files on several different systems. That lead to some head scratching because I couldn't comprehend why there would be a difference on that last whitespace.

regularfry 6y ago

So, there is a direct effect of this that's really annoying when you bump into it. Docker images are tarballs, and the layer cache relies on how the tar file is constructed. The same layer content constructed by two different tar implementations will appear different to the cache lookup. For reasons known only to themselves, Docker originally used different implementations for the docker command-line tool and the docker-compose tool. This means that cache entries produced by two tools that you got from the same place don't match up, and you can't use one tool to cache a layer and have the other pick it up. At least, not by default - if you know about it, there's a switch you can flip to have one use the other's implementation. But they had to add that in, and do that extra work, because tar isn't well-specified.

rozab 6y ago
Seems like gzip would be the bigger issue there. There are millions of valid ways to compress a single file