4 ms·

I always add an “Accept-Encoding” header to my request to indicate I will accept a gzip response (or deflate if available). Your http library (in whatever langu

I always add an “Accept-Encoding” header to my request to indicate I will accept a gzip response (or deflate if available). Your http library (in whatever language your bot is in) probably supports this with a near trivial amount of additional code, if any. Meanwhile you are saving the target site some bandwidth. Look into If-Modified-Since and If-None-Match/Etag headers as well if you are querying resources that support those headers (RSS feeds, for example, commonly support these, and static resources). They prevent the target site from having to send anything other than a 304, saving bandwidth and possibly compute.

Lammy 7y ago
> Meanwhile you are saving the target site some bandwidth. And costing them some CPU :) It’s probably a good idea in most cases, agreed, but there are exceptions such as if you are requesting resources in already-compressed formats, like most image/video codecs.
- deleted 7y ago
  [deleted]
- edaemon 7y ago
  The server ultimately decides what encoding gets used, so if the CPU cost is too high they can ignore the compression.
- newscracker 7y ago
  This is baffling to me, since I’ve always thought of gzip (or other) compression being applied by the web server (or configured to do so) only to text formats like HTML, JS, CSS, etc. I’m curious to know which badly written server or sites compress already compressed content like images and videos just because a user agent says it’ll accept compressed content?
- kerkeslager 7y ago
  Frankly, it would be difficult to find a part of your post that is correct. 1. You're never causing their server to do anything they didn't configure their server to do. Accept headers are merely information for the server telling them what you can accept: what they return to you is their choice, and they can weigh the tradeoffs themselves. 2. The tradeoff you think is happening isn't even happening in a lot of cases. In a lot of cases they'll be serving that up from a cache of some sort so the CPU work has already been done when someone else requested the page. CPU versus bandwidth isn't an inherent tradeoff.
- snuxoll 7y ago
  > And costing them some CPU On the servers that have no purpose in life but to handle caching. I’d much rather browsers and scrapers alike hit my Apache Trafficserver instances with requests needing to return a Not Modified than wasting time of the app servers.

Lammy 7y ago

> Meanwhile you are saving the target site some bandwidth. And costing them some CPU :) It’s probably a good idea in most cases, agreed, but there are exceptions such as if you are requesting resources in already-compressed formats, like most image/video codecs.

deleted 7y ago
[deleted]
edaemon 7y ago
The server ultimately decides what encoding gets used, so if the CPU cost is too high they can ignore the compression.
newscracker 7y ago
This is baffling to me, since I’ve always thought of gzip (or other) compression being applied by the web server (or configured to do so) only to text formats like HTML, JS, CSS, etc. I’m curious to know which badly written server or sites compress already compressed content like images and videos just because a user agent says it’ll accept compressed content?
kerkeslager 7y ago
Frankly, it would be difficult to find a part of your post that is correct. 1. You're never causing their server to do anything they didn't configure their server to do. Accept headers are merely information for the server telling them what you can accept: what they return to you is their choice, and they can weigh the tradeoffs themselves. 2. The tradeoff you think is happening isn't even happening in a lot of cases. In a lot of cases they'll be serving that up from a cache of some sort so the CPU work has already been done when someone else requested the page. CPU versus bandwidth isn't an inherent tradeoff.
snuxoll 7y ago
> And costing them some CPU On the servers that have no purpose in life but to handle caching. I’d much rather browsers and scrapers alike hit my Apache Trafficserver instances with requests needing to return a Not Modified than wasting time of the app servers.