4 ms·

Using Zip archives on Unix doesn't feel right, but it can solve a bunch of these problems straight away, while remaining highly portable.

Aloha 8y ago
How do zip archives solve these problems?
- cyphar 8y ago
  zip can solve one of the problems I listed, namely the parallel operations problem, because it has an index at the end of the archive which would facilitate random-access (though because it's at the end you'd have to download it fully before you could parallel-extract -- so it's only partially ideal). I wouldn't say zip can solve all of them though.
  - jacques_chester 8y ago
    The index at the end of the file is definitely a serious drawback for zipfiles; it privileges the writer over the reader (it makes sense given the hardware limitations at the time). For someone archiving their own stuff that's neither here nor there. For container images, JAR files etc the ratio is lopsided.
naniwaduni 8y ago
But why doesn't it feel right? Is it the lack of tooling with the Unix nature? (libarchive's tar will sort of do, although the zip files it creates are a bit wacky.)
- thayne 8y ago
  > But why doesn't it feel right Because it wasn't designed for for unix-like systems. Perhaps most notably, zip files don't have any support for unix-style permissions. Which makes it a bit of a non-starter for containers at least.
  - naniwaduni 8y ago
    They do---as an optional extra field. This seems reasonable; it's not even desirable for many archives to include e.g. owner/group information. The zip format has its own odd required metadata (a timestamp in local time!), but for the most part the design of relegating platform-specific metadata to the extra fields seems quite sensible. -UNIX Extra Field (0x000d): The following is the layout of the Unix "extra" block. Note: all fields are stored in Intel low-byte/high-byte order. Value Size Description ----- ---- ----------- (UNIX) 0x000d 2 bytes Tag for this "extra" block type TSize 2 bytes Size for the following data block Atime 4 bytes File last access time Mtime 4 bytes File last modification time Uid 2 bytes File user ID Gid 2 bytes File group ID (var) variable Variable length data field The variable length data field will contain file type specific data. Currently the only values allowed are the original "linked to" file names for hard or symbolic links, and the major and minor device node numbers for character and block device nodes. Since device nodes cannot be either symbolic or hard links, only one set of variable length data is stored. Link files will have the name of the original file stored. This name is NOT NULL terminated. Its size can be determined by checking TSize - 12. Device entries will have eight bytes stored as two 4 byte entries (in little endian format). The first entry will be the major device number, and the second the minor device number. http://www.pkware.com/documents/APPNOTE/APPNOTE_6.2.0.txt http://www.pkware.com/documents/APPNOTE/APPNOTE_6.2.0.txt
sitkack 8y ago
NIH? Almost all modern file formats are zip with something machine and humanish readable inside (XML, json, etc). OpenOffice, MS Office, jar, apk, epub, etc are all just zip files.
mwcampbell 8y ago
How about 7z? How do 7z, zip, and tar compare in terms of support for Unixy things like permissions, uid/gid, xattrs, and so on?
- chungy 8y ago
  7z only supports DOS-style attributes, while Zip supports the full host of Unix metadata like tar does.
  - jabl 8y ago
    IIRC the unix metadata is not part of the canonical zip format, but rather an extension introduced by (?) infozip, an implementation of zip for Unix systems (if you use the zip/unzip cli tools on a Unix style system, it's most likely infozip).

Aloha 8y ago

How do zip archives solve these problems?

cyphar 8y ago
zip can solve one of the problems I listed, namely the parallel operations problem, because it has an index at the end of the archive which would facilitate random-access (though because it's at the end you'd have to download it fully before you could parallel-extract -- so it's only partially ideal). I wouldn't say zip can solve all of them though.
- jacques_chester 8y ago
  The index at the end of the file is definitely a serious drawback for zipfiles; it privileges the writer over the reader (it makes sense given the hardware limitations at the time). For someone archiving their own stuff that's neither here nor there. For container images, JAR files etc the ratio is lopsided.

naniwaduni 8y ago

But why doesn't it feel right? Is it the lack of tooling with the Unix nature? (libarchive's tar will sort of do, although the zip files it creates are a bit wacky.)

thayne 8y ago
> But why doesn't it feel right Because it wasn't designed for for unix-like systems. Perhaps most notably, zip files don't have any support for unix-style permissions. Which makes it a bit of a non-starter for containers at least.
- naniwaduni 8y ago
  They do---as an optional extra field. This seems reasonable; it's not even desirable for many archives to include e.g. owner/group information. The zip format has its own odd required metadata (a timestamp in local time!), but for the most part the design of relegating platform-specific metadata to the extra fields seems quite sensible. -UNIX Extra Field (0x000d): The following is the layout of the Unix "extra" block. Note: all fields are stored in Intel low-byte/high-byte order. Value Size Description ----- ---- ----------- (UNIX) 0x000d 2 bytes Tag for this "extra" block type TSize 2 bytes Size for the following data block Atime 4 bytes File last access time Mtime 4 bytes File last modification time Uid 2 bytes File user ID Gid 2 bytes File group ID (var) variable Variable length data field The variable length data field will contain file type specific data. Currently the only values allowed are the original "linked to" file names for hard or symbolic links, and the major and minor device node numbers for character and block device nodes. Since device nodes cannot be either symbolic or hard links, only one set of variable length data is stored. Link files will have the name of the original file stored. This name is NOT NULL terminated. Its size can be determined by checking TSize - 12. Device entries will have eight bytes stored as two 4 byte entries (in little endian format). The first entry will be the major device number, and the second the minor device number. http://www.pkware.com/documents/APPNOTE/APPNOTE_6.2.0.txt http://www.pkware.com/documents/APPNOTE/APPNOTE_6.2.0.txt

sitkack 8y ago

NIH? Almost all modern file formats are zip with something machine and humanish readable inside (XML, json, etc). OpenOffice, MS Office, jar, apk, epub, etc are all just zip files.

mwcampbell 8y ago

How about 7z? How do 7z, zip, and tar compare in terms of support for Unixy things like permissions, uid/gid, xattrs, and so on?

chungy 8y ago
7z only supports DOS-style attributes, while Zip supports the full host of Unix metadata like tar does.
- jabl 8y ago
  IIRC the unix metadata is not part of the canonical zip format, but rather an extension introduced by (?) infozip, an implementation of zip for Unix systems (if you use the zip/unzip cli tools on a Unix style system, it's most likely infozip).