3 ms·
You can use varnish to cache all of these requests and just never expire them. Make the page as small as possible. It'll make the overhead of serving the requ
by bretpiatt 16y ago
You can use varnish to cache all of these requests and just never expire them. Make the page as small as possible. It'll make the overhead of serving the requests as low as possible.
http://www.varnish-cache.org/ http://www.varnish-cache.org/
- buro9 16y agoYou can also use Varnish ACLs to block the EC2 IP addresses. And if they have a really obvious user-agent you could block that using Varnish: http://omninoggin.com/web-development/block-unwanted-spam-bots-using-varnish-vcl/ http://omninoggin.com/web-development/block-unwanted-spam-bo... And if you're not afraid of using Varnish's VCL InlineC stuff, then you could add rate limiting to Varnish: http://drcarter.info/2010/04/how-fighting-against-scraping-using-varnish-vcl-inline-c-memcached/ http://drcarter.info/2010/04/how-fighting-against-scraping-u... Basically... Varnish has become my Swiss army tool for rejecting crap traffic (and caching good traffic obviously).