5 ms·

Please, can we move to an archive format that isn't so sprawlingly massive ?

by stuaxo 8y ago

spookthesunset 8y ago
What do you mean by massive?
cpburns2009 8y ago
Are you complaining about the complexity of file format itself? My understanding is it's pretty simple: a linked list of headers with the contents of each file after each header. Or are you complaining that it doesn't do compression itself like ZIPs do?
- GrayShade 8y ago
  One think I dislike about tarballs is the lack of random access support.
  - cpuguy83 8y ago
    You can build random access around tarballs, just need to index the header data. Shameless plug, this is what https://github.com/cpuguy83/tarfs https://github.com/cpuguy83/tarfs does. Granted, you do have to traverse the entire tarball.
    - masklinn 8y ago
      > You can build random access around tarballs, just need to index the header data. > Granted, you do have to traverse the entire tarball. So you can't randomly access a tarball, you can cache the linear access you've already done.
      - Vendan 8y ago
        You can read all the headers without reading the whole file by just seeking over the file data....
        sitkack 8y ago
        But you would still have to decompress it. How much support for legacy systems do we need vs just making a slightly better version of a jar?
        cpuguy83 8y ago
        Decompression has nothing to do with tar, though. But I agree, it is painful to deal with tar+gz.
        sitkack 8y ago
        > Decompression has nothing to do with tar Which is exactly the problem. Same issue occurred via volume and filesystem management resulting in ZFS. We need systems that compose and also elide. So something that both archives as set of files and compresses them w/o losing affordances over the layer below it in the process.
- TylerE 8y ago
  He's complaining about the simplicity. He wants something with less suck.
peterwwillis 8y ago
Or one that can list/extract files without reading the entire archive, or one that can use binary diffs, or one that supports encryption, or one that supports long file names, or one that isn't hamstrung by different implementations of different standards on different platforms, or one that doesn't use 512 byte blocks, or one that is actually usable on modern operating systems, ....
- sitkack 8y ago
  That time is now [0]. [0] https://www.sqlite.org/sar/doc/trunk/README.md https://www.sqlite.org/sar/doc/trunk/README.md
  - JdeBP 8y ago
    Actually, that time was 1986, if one excludes encryption (which is easy to add). * https://www.rpi.edu/dept/acm/packages/zoo/2.1/rs_aix31/src/zoo.h https://www.rpi.edu/dept/acm/packages/zoo/2.1/rs_aix31/src/z...
  - peterwwillis 8y ago
    > This program (named "sqlar") operates much like "zip", except that the compressed archive it builds is stored in an SQLite database > The motivation for this is to see how much larger an SQLite database file is compared to a ZIP archive containing the same content. The answer depends on the filenames, but 2% seems to be a reasonable guess. In other words, storing files as compressed blobs in an SQLite database file results in a file that is only about 2% larger than storing those same files in a ZIP archive using the same compression. Uh.... Yeah, I don't need a complicated, incompatible version of Zip that is 2% larger. I'll just use Zip.
rekado 8y ago
Sure. `guix pack` is a neat hack and it isn't tied to any particular archive format. When using plain Guix you won't need to use any archive format at all; packages simply end up each in their own unique directory and can be used just like that. You can easily spawn a container environment where only the relevant directories under `/gnu/store` are mounted. It's on my list to add more target formats for `guix pack`, but generally I'd recommend using Guix directly to reap all benefits. `guix pack` is only really useful for cases where you cannot use Guix on the target system.
- rekado 8y ago
  A squashfs backend for `guix pack` exists now: http://lists.gnu.org/archive/html/guix-patches/2018-05/msg00577.html http://lists.gnu.org/archive/html/guix-patches/2018-05/msg00...

Please, can we move to an archive format that isn't so sprawlingly massive ?

by stuaxo 8y ago

Please, can we move to an archive format that isn't so sprawlingly massive ?

spookthesunset 8y ago

What do you mean by massive?

cpburns2009 8y ago

Are you complaining about the complexity of file format itself? My understanding is it's pretty simple: a linked list of headers with the contents of each file after each header. Or are you complaining that it doesn't do compression itself like ZIPs do?

GrayShade 8y ago
One think I dislike about tarballs is the lack of random access support.
- cpuguy83 8y ago
  You can build random access around tarballs, just need to index the header data. Shameless plug, this is what https://github.com/cpuguy83/tarfs https://github.com/cpuguy83/tarfs does. Granted, you do have to traverse the entire tarball.
  - masklinn 8y ago
    > You can build random access around tarballs, just need to index the header data. > Granted, you do have to traverse the entire tarball. So you can't randomly access a tarball, you can cache the linear access you've already done.
    - Vendan 8y ago
      You can read all the headers without reading the whole file by just seeking over the file data....
      - sitkack 8y ago
        But you would still have to decompress it. How much support for legacy systems do we need vs just making a slightly better version of a jar?
        cpuguy83 8y ago
        Decompression has nothing to do with tar, though. But I agree, it is painful to deal with tar+gz.
        sitkack 8y ago
        > Decompression has nothing to do with tar Which is exactly the problem. Same issue occurred via volume and filesystem management resulting in ZFS. We need systems that compose and also elide. So something that both archives as set of files and compresses them w/o losing affordances over the layer below it in the process.
TylerE 8y ago
He's complaining about the simplicity. He wants something with less suck.

peterwwillis 8y ago

Or one that can list/extract files without reading the entire archive, or one that can use binary diffs, or one that supports encryption, or one that supports long file names, or one that isn't hamstrung by different implementations of different standards on different platforms, or one that doesn't use 512 byte blocks, or one that is actually usable on modern operating systems, ....

sitkack 8y ago
That time is now [0]. [0] https://www.sqlite.org/sar/doc/trunk/README.md https://www.sqlite.org/sar/doc/trunk/README.md
- JdeBP 8y ago
  Actually, that time was 1986, if one excludes encryption (which is easy to add). * https://www.rpi.edu/dept/acm/packages/zoo/2.1/rs_aix31/src/zoo.h https://www.rpi.edu/dept/acm/packages/zoo/2.1/rs_aix31/src/z...
- peterwwillis 8y ago
  > This program (named "sqlar") operates much like "zip", except that the compressed archive it builds is stored in an SQLite database > The motivation for this is to see how much larger an SQLite database file is compared to a ZIP archive containing the same content. The answer depends on the filenames, but 2% seems to be a reasonable guess. In other words, storing files as compressed blobs in an SQLite database file results in a file that is only about 2% larger than storing those same files in a ZIP archive using the same compression. Uh.... Yeah, I don't need a complicated, incompatible version of Zip that is 2% larger. I'll just use Zip.

rekado 8y ago

Sure. `guix pack` is a neat hack and it isn't tied to any particular archive format. When using plain Guix you won't need to use any archive format at all; packages simply end up each in their own unique directory and can be used just like that. You can easily spawn a container environment where only the relevant directories under `/gnu/store` are mounted. It's on my list to add more target formats for `guix pack`, but generally I'd recommend using Guix directly to reap all benefits. `guix pack` is only really useful for cases where you cannot use Guix on the target system.

rekado 8y ago
A squashfs backend for `guix pack` exists now: http://lists.gnu.org/archive/html/guix-patches/2018-05/msg00577.html http://lists.gnu.org/archive/html/guix-patches/2018-05/msg00...