3 ms·
Ask HN: Should we block non-gzip HTTP bots to reduce bandwidth costs?
I run web servers on EC2 (behind ELB) and dealing with high bandwidth charges.
The bulk of the bandwidth charges are due to bots scraping/crawling/pinging without any form of compression.
Is it safe to block requests made without the accept-encoding header? (and what HTTP status code should be used? 400?) Naively it seems it would only be inconveniencing poorly written and uninvited bots (SEO spam, bots trying to find exploits etc.)
Or respond with gzip compression, no matter what?
Or just stop using AWS since the bandwidth is too expensive because of situations like this?