Most compression algorithms have a limited data reference window (dictionary size) - tens to hundreds of MB being typical upper limits. If duplicated data is fu

Most compression algorithms have a limited data reference window (dictionary size) - tens to hundreds of MB being typical upper limits. If duplicated data is further apart, it gets compressed again. If the data isn't pre-compressed, or doesn't otherwise significantly vary, then using the right tools should de-duplicate the data. Things like lrzip, rsync batch mode, bdiff should work well. If large blocks are identical, many filesystem archivers/backup solutions should do good deduplication, and squashfs can be kind of nice.