8 ms·

I like simple archives, but can it be not tarballs? For the kinds of application described in this article, tarballs are pretty bad: Either you extract it from

by theamk 9y ago

I like simple archives, but can it be not tarballs? For the kinds of application described in this article, tarballs are pretty bad: Either you extract it from scratch every time you run an app, taking a long time penalty... ... or you extract once to cache, and assume that nothing changes the cache. This is pretty bad from both operational and security perspective: - backups have to walk through tens of thousands of files, thus becoming much slower - a damaged disk or a malicious actor can change one file in the cache, making damage which is very hard to detect. There are plenty of mountable container formats -- ISO, squashfs, even zip files -- which all provide much faster initial access, and much better security/reliability guarantees, especially with things like dm-verity.

2ion 9y ago
Yes, most tarballs do not support random access (there are some metadata extensions that allow this). This makes large tarballs annoying to use on systems with slow disk I/O (even a hard disk may be too slow (to the degree of being annoying to work with)). This is by far my biggest gripe with the format. Certainly, smaller tarballs are a very handy format as long as you stay inside the Unixy world of computing – and as long as you keep looking out for the various incompatibilities between the different tar implementations.
- textmode 9y ago
  "... there are some metadata extensions that allow this)." Where to find these extensions? Are they portable between Linux and BSD? The 1998 dict project included a utility called "dictzip" for random access to the contents of gzip compressed files. Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives? Example use case: the user only wants to untar a small number of selected files from a large tarball such as a source tree. The user has tried both the "-T filelist" option and using memory file systems instead of hard disk drives.
  - Hello71 9y ago
    afaik, only pixz does this; they store the index in xz.
  - barrkel 9y ago
    A zip file is a concatenation of gzipped files. A .tar.gz is a gzip stream of concatenated files. Anything that could do random access into the contents of a zip file entry could do similar things with a tarball.
    - jahewson 9y ago
      Not so simple. A bundle of streams is not the same as a stream of bundles.
      - sakuronto 9y ago
        What about .gz.tar, in which all the files are gzipped, then tarballed? It seems like it would be a slightly fatter .zip file.
      - barrkel 9y ago
        With a transparent random access overlay, the difference mostly disappears, reducing to whether the stream needs to be scanned or whether it's indexed, which is itself orthogonal - zip file directory at the end is redundant.
        netheril96 9y ago
        So you mean at each "random access", you actually have to scan the whole .tar.gz file to find the location? For large tarballs, that will definitely hinder performance a lot. The difference does not disappear at all.
        barrkel 9y ago
        Apparently you have a comprehension problem.
- pif 9y ago
  > This makes large tarballs annoying to use on systems with slow disk I/O Funny how tar was originally developed for tape drives!
  - coldacid 9y ago
    Tape drives don't really support random access, though, which is reflected in the design of the tar format and its offspring. That is, in fact, the problem here, and why formats designed for random access instead of sequential access are far better for storing file systems for containers and VMs.
RX14 9y ago
I'm pretty sure the article implies this is for user-facing applications where the user would manually extract it once to a place of their choosing then run it from there. I think you're missing the point of the whole article.
- theamk 9y ago
  But why would you want to extract if you can mount the file directly? For simple archives, extracting is fine. But for larger archives (like a compiler -- 1000 files or more), loop-mounting is much better than extracting: - Does not slow down your backup by adding thousands of files - No need to wait for initial file extraction - You can quickly and easily verify integrity of the whole archive And if you are using fuse, it does not require any special privileges either!
justincormack 9y ago
Mountable formats have the security issue that the kernel is not that great at protecting against hostile images in mount. On disk format fuzzing has not been common and there are definitely bugs.
- theamk 9y ago
  This is solved very nicely with fuse mounts (and fuse is surprisingly performant on the modern multicore systems)
solatic 9y ago
Do you not still pay a significant performance penalty by reverifying the container upon each application load? Especially considering that, if the container is signed, you need to verify the signature itself before trusting the container, and full signature verification - including checking whether the signature has been revoked - involves expensive network calls? If your operational and security model really frowns on trusting your extraction cache, then perhaps a different workflow is more appropriate - download the container, verify the container, extract, bake the OS plus extracted apps into an image, sign the image, verify the image upon each boot and mount apps read-only. Then you don't need to re-verify anything upon each launch, instead trusting that your image creation process is routinely updating and re-verifying the software in your current images.
- theamk 9y ago
  Verification of a single file is much faster than walking entire tree, especially when there are lots of small files, for example when there is a compiler or large python project inside. A simple example: my /usr/include is 33037 files, 356M uncompressed. On SSD with cold cache, it takes 6.7 sec to read each file individually, or 0.7 sec to checksum a single 356M archive, a 10x difference. The difference in the backup time is even more dramatic -- the backup program has to call stat() either 33K times, or just once, a 3,330,000% improvement! The other filesystem tools (What takes all the space? What has changed in the last X hours? Please sync this directory elsewhere.) will have similarly high speed improvements. So if I had a choice, I would love my dev environment to come in mountable form. Similarly, I don't understand why container runtimes (like docker) don't use loop mounts more -- it seems like many advantages and very few disadvantages. As for signature verification -- I don't care about 3rd party signature and revocation, I just want to ensure that I am running the same code every time. There are many ways one can damage extraction cache, especially if it is owned by the same user as application (like the topicstarter post described) -- sysadmin errors (`sudo find / -name app-old -delete`), application errors (create cache file in bin dir), disk errors (silent corruption), transfer errors (one file did not get transferred to a new computer). Loop mounting makes disk errors easier to detect, and eliminates other classes of error entirely.

by theamk 9y ago

2ion 9y ago

Yes, most tarballs do not support random access (there are some metadata extensions that allow this). This makes large tarballs annoying to use on systems with slow disk I/O (even a hard disk may be too slow (to the degree of being annoying to work with)). This is by far my biggest gripe with the format. Certainly, smaller tarballs are a very handy format as long as you stay inside the Unixy world of computing – and as long as you keep looking out for the various incompatibilities between the different tar implementations.

textmode 9y ago
"... there are some metadata extensions that allow this)." Where to find these extensions? Are they portable between Linux and BSD? The 1998 dict project included a utility called "dictzip" for random access to the contents of gzip compressed files. Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives? Example use case: the user only wants to untar a small number of selected files from a large tarball such as a source tree. The user has tried both the "-T filelist" option and using memory file systems instead of hard disk drives.
- Hello71 9y ago
  afaik, only pixz does this; they store the index in xz.
- barrkel 9y ago
  A zip file is a concatenation of gzipped files. A .tar.gz is a gzip stream of concatenated files. Anything that could do random access into the contents of a zip file entry could do similar things with a tarball.
  - jahewson 9y ago
    Not so simple. A bundle of streams is not the same as a stream of bundles.
    - sakuronto 9y ago
      What about .gz.tar, in which all the files are gzipped, then tarballed? It seems like it would be a slightly fatter .zip file.
    - barrkel 9y ago
      With a transparent random access overlay, the difference mostly disappears, reducing to whether the stream needs to be scanned or whether it's indexed, which is itself orthogonal - zip file directory at the end is redundant.
      - netheril96 9y ago
        So you mean at each "random access", you actually have to scan the whole .tar.gz file to find the location? For large tarballs, that will definitely hinder performance a lot. The difference does not disappear at all.
        barrkel 9y ago
        Apparently you have a comprehension problem.
pif 9y ago
> This makes large tarballs annoying to use on systems with slow disk I/O Funny how tar was originally developed for tape drives!
- coldacid 9y ago
  Tape drives don't really support random access, though, which is reflected in the design of the tar format and its offspring. That is, in fact, the problem here, and why formats designed for random access instead of sequential access are far better for storing file systems for containers and VMs.

RX14 9y ago

I'm pretty sure the article implies this is for user-facing applications where the user would manually extract it once to a place of their choosing then run it from there. I think you're missing the point of the whole article.

theamk 9y ago
But why would you want to extract if you can mount the file directly? For simple archives, extracting is fine. But for larger archives (like a compiler -- 1000 files or more), loop-mounting is much better than extracting: - Does not slow down your backup by adding thousands of files - No need to wait for initial file extraction - You can quickly and easily verify integrity of the whole archive And if you are using fuse, it does not require any special privileges either!

justincormack 9y ago

Mountable formats have the security issue that the kernel is not that great at protecting against hostile images in mount. On disk format fuzzing has not been common and there are definitely bugs.

theamk 9y ago
This is solved very nicely with fuse mounts (and fuse is surprisingly performant on the modern multicore systems)

solatic 9y ago

Do you not still pay a significant performance penalty by reverifying the container upon each application load? Especially considering that, if the container is signed, you need to verify the signature itself before trusting the container, and full signature verification - including checking whether the signature has been revoked - involves expensive network calls? If your operational and security model really frowns on trusting your extraction cache, then perhaps a different workflow is more appropriate - download the container, verify the container, extract, bake the OS plus extracted apps into an image, sign the image, verify the image upon each boot and mount apps read-only. Then you don't need to re-verify anything upon each launch, instead trusting that your image creation process is routinely updating and re-verifying the software in your current images.

theamk 9y ago
Verification of a single file is much faster than walking entire tree, especially when there are lots of small files, for example when there is a compiler or large python project inside. A simple example: my /usr/include is 33037 files, 356M uncompressed. On SSD with cold cache, it takes 6.7 sec to read each file individually, or 0.7 sec to checksum a single 356M archive, a 10x difference. The difference in the backup time is even more dramatic -- the backup program has to call stat() either 33K times, or just once, a 3,330,000% improvement! The other filesystem tools (What takes all the space? What has changed in the last X hours? Please sync this directory elsewhere.) will have similarly high speed improvements. So if I had a choice, I would love my dev environment to come in mountable form. Similarly, I don't understand why container runtimes (like docker) don't use loop mounts more -- it seems like many advantages and very few disadvantages. As for signature verification -- I don't care about 3rd party signature and revocation, I just want to ensure that I am running the same code every time. There are many ways one can damage extraction cache, especially if it is owned by the same user as application (like the topicstarter post described) -- sysadmin errors (`sudo find / -name app-old -delete`), application errors (create cache file in bin dir), disk errors (silent corruption), transfer errors (one file did not get transferred to a new computer). Loop mounting makes disk errors easier to detect, and eliminates other classes of error entirely.