Back when I was involved, the compression was gzip -- each archived page (HTTP headers + body) was separately gzipped and concatenated onto a ~100M file, though

Back when I was involved, the compression was gzip -- each archived page (HTTP headers + body) was separately gzipped and concatenated onto a ~100M file, though I believe they're now doing 1G files. (This cleverly allows retrieving the individual document by seeking to that offset and doing gzip decompression.) They have significant amounts of inline images and other files in the web archive, which of course typically are already compressed. http://en.wikipedia.org/wiki/Web_ARChive http://en.wikipedia.org/wiki/Web_ARChive