Two points worth noting: 1. Gzip is not a suitable compressor for this use case, because it's limited to a 32KB window. So the input can only be correlated wit

Two points worth noting: 1. Gzip is not a suitable compressor for this use case, because it's limited to a 32KB window. So the input can only be correlated with the last 32KB of the reference texts. 2. You can save a great deal in computation by avoiding recompressing the reference texts over and over and over. Some compression algorithms support checkpointing the compression state so that it can be resumed from that point repeatedly ("dictionary-based compression", which is a distinct capability from just streaming compression, which generally can only be continued once). I would personally shill for using Zstandard [0] instead for this purpose. Although I should disclose my bias: I'm a developer of Zstd. A few salient facts: 1. Zstd supports very large windows (up to 128MB, or up to 2GB in long mode). 2. Zstd is much faster than zlib. 3. Zstd has well-developed support for dictionary-based compression. 4. Additionally, it has a dictionary trainer that can reduce a corpus of reference documents to a compact summary document that aims to capture as much of the content as possible of the reference corpus. [1] 5. It has (more than one) python binding available. [2][3] [0] https://github.com/facebook/zstd https://github.com/facebook/zstd [1] https://github.com/facebook/zstd/blob/dev/lib/zdict.h#L40 https://github.com/facebook/zstd/blob/dev/lib/zdict.h#L40 [2] https://pypi.org/project/zstandard https://pypi.org/project/zstandard [3] https://pypi.org/project/zstd https://pypi.org/project/zstd