5 ms·

There exists a utility called tarindexer [0] that can be used for random access to tar files. An index text file is created (one time) that is used to record th

There exists a utility called tarindexer [0] that can be used for random access to tar files. An index text file is created (one time) that is used to record the position of the files in the tar archive. Random reads are done by loading the index file and then seeking to the location of the file in question. For random access to gzip'd files, bgzip [1] can be used. bgzip also uses an index file (one time creation) that is used to record key points for random access. [0] https://github.com/devsnd/tarindexer https://github.com/devsnd/tarindexer [1] http://www.htslib.org/doc/bgzip.html http://www.htslib.org/doc/bgzip.html

selfhoster11 5y ago
That just sounds like an inferior version of ZIP. IMO, unless you can only work with the tar format (which is a perfectly valid explanation, some programs are long-lived), ZIP is a better option for seekable archives because it supports even LZMA compression.
- lazide 5y ago
  Zip is….. weird. In some undesirably ways sometimes, and desirable in others. The index is stored at the tail end of a zip file for instance, which is really not cool for something likej tape, doubly not cool when you don’t know in advance how big the data on the tape is.
  - selfhoster11 5y ago
    Don't tapes support some form of seeking?
    - lazide 5y ago
      They do - it’s very slow, and entirely linear. It also puts wear on the tape, so if you do it a lot you’ll break the tape in a not-super-long-time. And since you wouldn’t know where the end is by reading the beginning, you’ll have to keep reading until you hit the right marker - and seek backwards (which you generally can’t read backwards on most tape drives, so you need to jump back and re-read).
    - Isthatablackgsd 5y ago
      What lazide saying is that some important information is stored at the end (as in the very end of the book/movie or at the end of the line). So imagine there is a 1TB .zip archive in the tape, the tape device have to go further (deeper) into that 1TB file to get that last bit of vital data that the user want to see. Normally the vital bit usually at the start of the file (as in the front line/beginning of the book) that the user have the information ready before they could transfer it. But for lazide case, the tape device have to keep reading the entire 1TB zip to get that last vital bit of information which made it slow. It is more like it cannot "skip the line" and would have to go through entire line to get there.
  - larkost 5y ago
    It is a little more complicated than that: having the index at the end is great for writing tape, but sucks for reading tape. The nice thing of that design is that you can "stream" the data (read one file, write it, and just make some notes on the side for your eventual index). But you can't stream things off (you have to read the index first). Tar is of course all-around great for tape, since every file is essentially put on by itself (with just a little bit of header about it). Again this is great for streaming things both on and off tape. But you can't do any sort of jumping around. You have to start at the beginning and then go from file to file. This gets even worse if you try to compress it (e.g. .tar.gz), as you then have to decompress every byte to get to the next one.
    - tarasglek 5y ago
      couple of points: a) zip spec does not require file index to be at the end. Only the pointer to file index b) You can do streaming decompression of zip as each zip entry has a header + filename inline with data. c) I came up with this optimization(placing index at front of making file io sequential to make use of OS readahead) for omni.ja files in firefox. It's still a standard zip file, but index lives at the front. Most zip tools can open that file unmodified(tho they sometimes complain).
      - lazide 5y ago
        Ho, learn something new everyday, thanks! Also, for anyone else reading this found this elsewhere and is applicable [https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT] That is actually applicable to a project I’m working on now, any chance your implementation is in python and open source somewhere?
        tarasglek 5y ago
        Yes, it's in python :) https://github.com/humphd/mozilla-central-old/blob/9d4d9f265e24e6358c067ae1e300c1ce3227a91d/config/optimizejars.py https://github.com/humphd/mozilla-central-old/blob/9d4d9f265... Curious, how is it applicable?
  - petre 5y ago
    Also limited to 4Gb unless it's zip64 which is limited to 16 Eb and not supported by all zip implementations.
cb321 5y ago
Also relevant is pixz [1] which can do parallel LZMA/XZ decompression as well as tar file indexing. [1] https://github.com/vasi/pixz https://github.com/vasi/pixz
zinconfire 5y ago
I've done the same thing for decades with the -v -R options to GNU tar, which cause it to print the block number of each file to stderr while creating the archive. With knowledge of the block number, you can then pipe dd to tar to extract a single file without a linear search. Or, in the case of tapes, you can use mt to seek to the block before you run tar. Of course, something like tarindexer is more convenient!
lathiat 5y ago
I've recently been looking into this same issue because I analyse a lot of data like sosreports or other tar/compressed data from customer systems. Currently I untar these onto my zfs filesystem which works out OK because it has zstd compression enabled but I end up decompressing and recompressing which is quite expensive as often the files are GBs or more compressed. But I've started using a tool called "ratarmount" (https://github.com/mxmlnkn/ratarmount https://github.com/mxmlnkn/ratarmount) which creates an index once (and something I could automate our upload system to generate in advance, but you can also just process it lcoally) and then lets you fuse mount the file. This works pretty great with the only exception that I can't create scratch files inside the directory layout which in the past I'd wanted to do. I was surprised how hard a problem to solve it is to get a bundle file format that is indexable and compressed with a good and fast compression algorithm which mostly boils down to zstd at this point. While it works quite well, especially with gzip and bzip2, sadly the zstd and xz (and some other compression formats) don't allow for decompressing only parts of a file by default, even though it's possible the default tools aren't doing it. The nitty gritty details are summarised here: https://github.com/mxmlnkn/ratarmount#xz-and-zst-files https://github.com/mxmlnkn/ratarmount#xz-and-zst-files The other main options I found was squashfs which recently grew zstd support and there is some preliminary zip file support for zstd but there are multiple standards which is not helpful!

selfhoster11 5y ago

That just sounds like an inferior version of ZIP. IMO, unless you can only work with the tar format (which is a perfectly valid explanation, some programs are long-lived), ZIP is a better option for seekable archives because it supports even LZMA compression.

lazide 5y ago
Zip is….. weird. In some undesirably ways sometimes, and desirable in others. The index is stored at the tail end of a zip file for instance, which is really not cool for something likej tape, doubly not cool when you don’t know in advance how big the data on the tape is.
- selfhoster11 5y ago
  Don't tapes support some form of seeking?
  - lazide 5y ago
    They do - it’s very slow, and entirely linear. It also puts wear on the tape, so if you do it a lot you’ll break the tape in a not-super-long-time. And since you wouldn’t know where the end is by reading the beginning, you’ll have to keep reading until you hit the right marker - and seek backwards (which you generally can’t read backwards on most tape drives, so you need to jump back and re-read).
  - Isthatablackgsd 5y ago
    What lazide saying is that some important information is stored at the end (as in the very end of the book/movie or at the end of the line). So imagine there is a 1TB .zip archive in the tape, the tape device have to go further (deeper) into that 1TB file to get that last bit of vital data that the user want to see. Normally the vital bit usually at the start of the file (as in the front line/beginning of the book) that the user have the information ready before they could transfer it. But for lazide case, the tape device have to keep reading the entire 1TB zip to get that last vital bit of information which made it slow. It is more like it cannot "skip the line" and would have to go through entire line to get there.
- larkost 5y ago
  It is a little more complicated than that: having the index at the end is great for writing tape, but sucks for reading tape. The nice thing of that design is that you can "stream" the data (read one file, write it, and just make some notes on the side for your eventual index). But you can't stream things off (you have to read the index first). Tar is of course all-around great for tape, since every file is essentially put on by itself (with just a little bit of header about it). Again this is great for streaming things both on and off tape. But you can't do any sort of jumping around. You have to start at the beginning and then go from file to file. This gets even worse if you try to compress it (e.g. .tar.gz), as you then have to decompress every byte to get to the next one.
  - tarasglek 5y ago
    couple of points: a) zip spec does not require file index to be at the end. Only the pointer to file index b) You can do streaming decompression of zip as each zip entry has a header + filename inline with data. c) I came up with this optimization(placing index at front of making file io sequential to make use of OS readahead) for omni.ja files in firefox. It's still a standard zip file, but index lives at the front. Most zip tools can open that file unmodified(tho they sometimes complain).
    - lazide 5y ago
      Ho, learn something new everyday, thanks! Also, for anyone else reading this found this elsewhere and is applicable [https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT] That is actually applicable to a project I’m working on now, any chance your implementation is in python and open source somewhere?
      - tarasglek 5y ago
        Yes, it's in python :) https://github.com/humphd/mozilla-central-old/blob/9d4d9f265e24e6358c067ae1e300c1ce3227a91d/config/optimizejars.py https://github.com/humphd/mozilla-central-old/blob/9d4d9f265... Curious, how is it applicable?
- petre 5y ago
  Also limited to 4Gb unless it's zip64 which is limited to 16 Eb and not supported by all zip implementations.

cb321 5y ago

Also relevant is pixz [1] which can do parallel LZMA/XZ decompression as well as tar file indexing. [1] https://github.com/vasi/pixz https://github.com/vasi/pixz

zinconfire 5y ago

I've done the same thing for decades with the -v -R options to GNU tar, which cause it to print the block number of each file to stderr while creating the archive. With knowledge of the block number, you can then pipe dd to tar to extract a single file without a linear search. Or, in the case of tapes, you can use mt to seek to the block before you run tar. Of course, something like tarindexer is more convenient!

lathiat 5y ago

I've recently been looking into this same issue because I analyse a lot of data like sosreports or other tar/compressed data from customer systems. Currently I untar these onto my zfs filesystem which works out OK because it has zstd compression enabled but I end up decompressing and recompressing which is quite expensive as often the files are GBs or more compressed. But I've started using a tool called "ratarmount" (https://github.com/mxmlnkn/ratarmount https://github.com/mxmlnkn/ratarmount) which creates an index once (and something I could automate our upload system to generate in advance, but you can also just process it lcoally) and then lets you fuse mount the file. This works pretty great with the only exception that I can't create scratch files inside the directory layout which in the past I'd wanted to do. I was surprised how hard a problem to solve it is to get a bundle file format that is indexable and compressed with a good and fast compression algorithm which mostly boils down to zstd at this point. While it works quite well, especially with gzip and bzip2, sadly the zstd and xz (and some other compression formats) don't allow for decompressing only parts of a file by default, even though it's possible the default tools aren't doing it. The nitty gritty details are summarised here: https://github.com/mxmlnkn/ratarmount#xz-and-zst-files https://github.com/mxmlnkn/ratarmount#xz-and-zst-files The other main options I found was squashfs which recently grew zstd support and there is some preliminary zip file support for zstd but there are multiple standards which is not helpful!