4 ms·

It is a little more complicated than that: having the index at the end is great for writing tape, but sucks for reading tape. The nice thing of that design is t

It is a little more complicated than that: having the index at the end is great for writing tape, but sucks for reading tape. The nice thing of that design is that you can "stream" the data (read one file, write it, and just make some notes on the side for your eventual index). But you can't stream things off (you have to read the index first). Tar is of course all-around great for tape, since every file is essentially put on by itself (with just a little bit of header about it). Again this is great for streaming things both on and off tape. But you can't do any sort of jumping around. You have to start at the beginning and then go from file to file. This gets even worse if you try to compress it (e.g. .tar.gz), as you then have to decompress every byte to get to the next one.

tarasglek 5y ago
couple of points: a) zip spec does not require file index to be at the end. Only the pointer to file index b) You can do streaming decompression of zip as each zip entry has a header + filename inline with data. c) I came up with this optimization(placing index at front of making file io sequential to make use of OS readahead) for omni.ja files in firefox. It's still a standard zip file, but index lives at the front. Most zip tools can open that file unmodified(tho they sometimes complain).
- lazide 5y ago
  Ho, learn something new everyday, thanks! Also, for anyone else reading this found this elsewhere and is applicable [https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT] That is actually applicable to a project I’m working on now, any chance your implementation is in python and open source somewhere?
  - tarasglek 5y ago
    Yes, it's in python :) https://github.com/humphd/mozilla-central-old/blob/9d4d9f265e24e6358c067ae1e300c1ce3227a91d/config/optimizejars.py https://github.com/humphd/mozilla-central-old/blob/9d4d9f265... Curious, how is it applicable?
    - lazide 5y ago
      Offline high speed data ingestion of multi-thousand file, multi hundred GB data sets, followed by rapid transfer to permanent online storage (and replication fan-out, etc). Seems convenient to allow optimization for high speed sequential reads and random read/writes at different parts of the life cycle, along with indexing, crcs, signatures, etc. One big issue with zip storing the index at the end of course is a truncated file basically lost most of it’s context and is generally unrecoverable even in part, which this could also help with from a durability perspective. Storing it at the beginning (without a end pointer) opens up the possibility you have a valid looking archive you’re touching that is truncated and missing a lot of data, and won’t know until you look past the end (or validate total bytes or whatever, which doesn’t work well when steaming). Storing the index at the beginning, pointer and file sig at the end, and all the other format extensions does solve for all this. Which is convenient.
      - tarasglek 5y ago
        neat, let me know if you have any further questions. Would love to make this a more common thing that happens to zip files. As result of me doing this in firefox, zip utilities(aka 7zip) started complaining a lot less about this creative interpretation of the standard :)

tarasglek 5y ago

couple of points: a) zip spec does not require file index to be at the end. Only the pointer to file index b) You can do streaming decompression of zip as each zip entry has a header + filename inline with data. c) I came up with this optimization(placing index at front of making file io sequential to make use of OS readahead) for omni.ja files in firefox. It's still a standard zip file, but index lives at the front. Most zip tools can open that file unmodified(tho they sometimes complain).

lazide 5y ago
Ho, learn something new everyday, thanks! Also, for anyone else reading this found this elsewhere and is applicable [https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT] That is actually applicable to a project I’m working on now, any chance your implementation is in python and open source somewhere?
- tarasglek 5y ago
  Yes, it's in python :) https://github.com/humphd/mozilla-central-old/blob/9d4d9f265e24e6358c067ae1e300c1ce3227a91d/config/optimizejars.py https://github.com/humphd/mozilla-central-old/blob/9d4d9f265... Curious, how is it applicable?
  - lazide 5y ago
    Offline high speed data ingestion of multi-thousand file, multi hundred GB data sets, followed by rapid transfer to permanent online storage (and replication fan-out, etc). Seems convenient to allow optimization for high speed sequential reads and random read/writes at different parts of the life cycle, along with indexing, crcs, signatures, etc. One big issue with zip storing the index at the end of course is a truncated file basically lost most of it’s context and is generally unrecoverable even in part, which this could also help with from a durability perspective. Storing it at the beginning (without a end pointer) opens up the possibility you have a valid looking archive you’re touching that is truncated and missing a lot of data, and won’t know until you look past the end (or validate total bytes or whatever, which doesn’t work well when steaming). Storing the index at the beginning, pointer and file sig at the end, and all the other format extensions does solve for all this. Which is convenient.
    - tarasglek 5y ago
      neat, let me know if you have any further questions. Would love to make this a more common thing that happens to zip files. As result of me doing this in firefox, zip utilities(aka 7zip) started complaining a lot less about this creative interpretation of the standard :)