Yes, it works with files produced by the usual gzip! That's why it is novel. If you make requirements on the compressor, then a multitude of competing gzip impl

Yes, it works with files produced by the usual gzip! That's why it is novel. If you make requirements on the compressor, then a multitude of competing gzip implementations exist already, the most popular of which, I think, is bgzip [0]. Rapidgzip can actually leverage the additional metadata stored in bgzip files to speed up decompression by roughly a factor of 2 compared to files compressed with the usual gzip. Rapidgzip does not require sync points because it simply looks for them. The approach is similar to gzrecover [1], i.e., you "simply" try to start decompression at every possible >bit< position. Of course that would be too slow (~200 kB/s), so instead I use several skip and lookup tables to speed up this search to achieve roughly (~70 MB/s). However, that search for deflate block boundaries can yield positions that look valid but which actually aren't, e.g., think of gzip-compressed gzip files. These cases also need to be detected and filtered. Rapidgzip does this filtering and the parallelization by using a cache which stores prefetched decompressed results with the start bit position as key. When the previous block has finished decoding, we know the start position of the next block, and if it exists in the cache, then that result is used. This implicitly filters out valid-looking duds, i.e., false positives. The proof of concept of looking for block boundaries and to resolve and parallelize interdependencies between consecutive blocks, was introduced by pugz [1]. However, it had no detection for false positives and therefore did only work for compressed text files. I built upon this to make it work without assumptions on the compressed data, and I also added capabilities for random access, for which the prefetch-cache architecture also is very suited, so that I can use it in ratarmount. [0] http://www.htslib.org/doc/bgzip.html http://www.htslib.org/doc/bgzip.html [1] https://github.com/arenn/gzrt https://github.com/arenn/gzrt [2] https://arxiv.org/abs/1905.07224 https://arxiv.org/abs/1905.07224