5 ms·
"Files that are sent to the client are compressed with gzip, unless they are under 4096 bytes." That's interesting. Is that a common optimization? I hadn't h
by tyingq 8y ago
"Files that are sent to the client are compressed with gzip, unless they are under 4096 bytes."
That's interesting. Is that a common optimization? I hadn't heard of any other web server doing that.
- hombre_fatal 8y agoI'd be surprised if it didn't exist in every compression middleware. For example, https://github.com/expressjs/compression/blob/dd5055dc92fdeacad706972c4fcf3a7ff10066ef/index.js#L55 https://github.com/expressjs/compression/blob/dd5055dc92fdea...
- tyingq 8y agoI don't see anything like that documented for apache's mod_deflate/zlib.
- hombre_fatal 8y agoSeems not, but there is for mod_gzip for example. I'd definitely consider it common.
- nnx 8y agoNginx uses a similar optimization with a configurable threshold defaulted to 1kb.
- nodesocket 8y agoCorrect, the NGINX configuration should be something like: gzip on; gunzip on; gzip_http_version 1.1; gzip_proxied any; gzip_comp_level 5; gzip_disable "msie6"; gzip_vary on; gzip_min_length 2048;
- arkadiyt 8y agoCloudfront is similar [1]: > The file size must be between 1,000 and 10,000,000 bytes. [1]: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
- DarkWiiPlayer 8y agoI heared somewhere (Some blogs comment section, I believe) that gzip actually reduces the security of HTTPS; maybe someone can confirm / explain that?
- tialaramex 8y agoCompression technologies, including gzip, obviously have the goal of making things smaller by predicting later data based on earlier data. If the later data looks more like the earlier data, the result is smaller than if it was random gibberish. Compression! If an attacker controls /some/ of this data, and would like to read /other parts/, they can abuse compression to measure whether the parts they don't know are "like" the part they control, because if they are then the compression will make the results shorter than otherwise which they can passively measure. It's not a problem to move a compressed object over a secure channel on its own, the problem arises if either you try to compress the channel which is moving objects from different origins (e.g. a cookie set by a random advertising web site and your Facebook password) or compress a composite object e.g. maybe your backups mixed with a file you downloaded from a dodgy "pirate" video site.
- takeda 8y agoIn scenario when an attacker can see encrypted message (e.g. monitoring network traffic) and can affect part of the message that is being encrypted, he can use the compression to his advantage. He can for example try different inputs and observe the length of encrypted message, if with the certain input the length drops that means the given input contains string that's similar to another part of the decrypted message and the compressing algorithm did its job and used that to reduce the size. This was mentioned in 2012 when CRIME[1] (also included BEAST[2] exploit) and later BREACH[2] vulnerability (when it was considered cool to come up with cool sounding name, creating a logo and a website for specific vulnerabilities) [1] https://en.wikipedia.org/wiki/CRIME https://en.wikipedia.org/wiki/CRIME [2] https://en.wikipedia.org/wiki/Transport_Layer_Security#BEAST_attack https://en.wikipedia.org/wiki/Transport_Layer_Security#BEAST... [3] https://en.wikipedia.org/wiki/BREACH https://en.wikipedia.org/wiki/BREACH
- emilfihlman 8y agoThis shouldn't be true in any sane system.
- Traubenfuchs 8y agoTomcat, one of the most used Java application servers and the default server behind Spring Boot applications defaults to a "compressionMinSize " value of 2048. The docs do not explain why.
- xyproto 8y agoI did some quick benchmarking and for files under roughly 4096 bytes, not compressing with gzip was faster. It's not terribly exact, +-1000 bytes would probably not make a big difference, but I think it's a good default. And of course, some people may have unique use cases where a custom threshold may be better.
- jand 8y agoNot a direct answer, but also interesting to consider: As everything will end up in a packet when sent through the network stack, you might want to choose your minimum input size in such way, that you generate gzip-compressed output big enough. Why big enought? Nagle's algorithm [1] So yet another reason to think about 'what to gzip'. [1] https://en.wikipedia.org/wiki/Nagle%27s_algorithm https://en.wikipedia.org/wiki/Nagle%27s_algorithm
- tialaramex 8y agoApplications that always know exactly what they want to send disable this algorithm, as that article explains, by setting TCP_NODELAY or its moral equvialents in their framework. A web server will almost invariably set TCP_NODELAY. More sophisticated algorithms can either decide exactly which packets to send, or use TCP_CORK to shove part of a packet into a buffer before they add the rest of the stuff, e.g. preparing HTTP headers and then adding the static document that goes after them.
- tedunangst 8y agoIf you read the page you linked, it says the algorithm applies to data of any size.
- codebeaker 8y agoAlso, according to the spec HTTP servers may not always honour the value in the `Accept-Encoding` header[0]. > Even if both the client and the server supports the same compression algorithms, the server may choose not to compress the body of a response, if the identity value is also acceptable. I've actually run into this twice in my career and it has been a surprise to those around me in both cases. Both times in the context of small payloads where the server is applying some heuristic about whether to encode or not. (e.g status page stops sending gzipped output when the server is becoming "unhealthy") [0]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ac...
- hombre_fatal 8y agoMakes more sense to use the verb "negotiating" with Accept-* headers rather than "honoring". This makes obvious sense once you consider that the client tells the server which compression formats it supports in every request yet not every data format is compressible, nor does the server necessary support any candidate compression format. For example, the server wouldn't gzip a jpeg since it's already compressed. All Accept-* headers are like this. e.g. the server doesn't necessarily support any of the languages requested in the Accept-Language header, but it doesn't hurt to ask. You always have to inspect the response headers to see the result of negotiation.
- takeda 8y ago"Accept-Encoding" means only that the client also understands specific encoding (in this case compression) it is still up to the server to chose what to dot. There was a time when browsers didn't support any compression. This header was introduced to signal to server what is acceptable by the client, that's why the header allows specifying multiple compression algorithms. Similar thing is with header that's quite useful, but for some reason very few sites honor it: "Accept-Language" browser can specify which languages are preferred, but it is up to server to honor it (for example given language version is not available).
- Malic 8y agoI don't know about "under 4096 bytes" but I have heard of not compressing data that is under ~1500 bytes. Part of the thinking is this - if your result data (plus HTTP overhead) is already smaller than the data payload of an IP packet (MTU settings come into play here), then you are spending CPU time that will not save you any network I/O time.
- xyproto 8y agoYes, also compressing and decompressing a small amount of bytes may take longer than just sending it uncompressed.