2 ms·

Compression is all about detecting patterns in the data, usually with some kind of sliding window of past data, used to predict future data. For gzip in particu

Compression is all about detecting patterns in the data, usually with some kind of sliding window of past data, used to predict future data. For gzip in particular, this sliding window has a fixed size of 32KB. If the "weird" version is ~2000× the size of the original source, gzip would need a sliding window of ~2000× the size (about 64MB) to obtain equivalent compression.

wizzard0 3y ago
Also, an encoder like these can inject any amount of entropy (eg replace "0" with "100500999-100500999") which gzip can reduce to "100500999-same" but without knowledge of JS semantics has no chance to reduce to either "0" nor the nearby "172913211-172913211")
josephg 3y ago
> For gzip in particular, this sliding window has a fixed size of 32KB. I'm surprised people keep reaching for gzip for this sort of comparison. LZ4 / zstd compress better at the same compression speed, and zstd and brotli compresses much better. Brotli is also supported in all modern browsers. There's basically no reason to use gzip in new software. (With the exception of compressing for the web - at least until zstd makes it into browsers.)
- roywashere 3y ago
  In this case, op explicitly compiled javascript for use in web browsers, so in my opinion reaching for gzip is not so strange. There is no browser support yet except behind feature flags in Chrome, and in curl, but that's not a browser. Also, zstd was explicitly designed to compress as good as deflate but much faster. You can also use zstd to compress 'better' but that costs some CPU time again but that is mostly a different use case
  - josephg 3y ago
    Brotli is a much better choice for compressing JavaScript for the web. It’s supported by every browser (and curl I think), and the compression ratio is way better. The only downside is that you probably want to compress once instead of on every request, because brotli compression is slow. Easy enough to set up though.
himinlomax 3y ago
It would be interesting to see what better compression schemes could do here.

wizzard0 3y ago

Also, an encoder like these can inject any amount of entropy (eg replace "0" with "100500999-100500999") which gzip can reduce to "100500999-same" but without knowledge of JS semantics has no chance to reduce to either "0" nor the nearby "172913211-172913211")

josephg 3y ago

> For gzip in particular, this sliding window has a fixed size of 32KB. I'm surprised people keep reaching for gzip for this sort of comparison. LZ4 / zstd compress better at the same compression speed, and zstd and brotli compresses much better. Brotli is also supported in all modern browsers. There's basically no reason to use gzip in new software. (With the exception of compressing for the web - at least until zstd makes it into browsers.)

roywashere 3y ago
In this case, op explicitly compiled javascript for use in web browsers, so in my opinion reaching for gzip is not so strange. There is no browser support yet except behind feature flags in Chrome, and in curl, but that's not a browser. Also, zstd was explicitly designed to compress as good as deflate but much faster. You can also use zstd to compress 'better' but that costs some CPU time again but that is mostly a different use case
- josephg 3y ago
  Brotli is a much better choice for compressing JavaScript for the web. It’s supported by every browser (and curl I think), and the compression ratio is way better. The only downside is that you probably want to compress once instead of on every request, because brotli compression is slow. Easy enough to set up though.

himinlomax 3y ago

It would be interesting to see what better compression schemes could do here.