3 ms·
Ask HN: Should we block non-gzip HTTP bots to reduce bandwidth costs?
We run web servers on EC2 (behind ELB) and are dealing with high bandwidth charges.
The bulk of the bandwidth charges are due to bots scraping/crawling/pinging without gzip.
Is it safe to block requests made without the accept-encoding header? (and what HTTP status code should be used? 400?) Naively it seems it would only be inconveniencing poorly written and uninvited bots (SEO spam, bots trying to find exploits etc.)
Or should we respond with gzip compression, no matter what?
Or should we stop using AWS since the bandwidth is too expensive because of situations like this?
- codegeek 9y agomay be consider adding something like cloudflare as a proxy to filter out the bots etc ? Only send what you need to AWS.