Indeed! Restarting at record boundaries is one option. Then an external index can point to all the ragged restart offsets, or some applications may be able to j

Indeed! Restarting at record boundaries is one option. Then an external index can point to all the ragged restart offsets, or some applications may be able to just scan for the distinct byte-patterns of likely restart offsets. Restarting at every N-bytes boundary – for example, every 128K – is another option, though that requires a little more fancy thought to padding. Another way to skip the need for an external index using any of these techniques, but still know the raw-data line/byte-offset when jumping to a deep position, is to encode those tallies (either chunk-by-chunk or from-start) as GZIP header "extra field" extensions. (Here's an example extension that just records per-chunk compressed- and uncompressed- byte-lengths: http://archive-access.sourceforge.net/warc/warc_file_format-0.9.html#anchor25 http://archive-access.sourceforge.net/warc/warc_file_format-...)