I don't block completely, but have for a long time configured servers to react like this: - If it's cached content, there's no limit - If it's non cached cont

I don't block completely, but have for a long time configured servers to react like this: - If it's cached content, there's no limit - If it's non cached content and it is a bot, direct to the pool of servers assigned to uninteresting bots - If it is an interesting bot (Google, bing, a couple others), use instead a pool of servers for interesting bots Some bots are not only non-traffic generating, they are absurdly aggressive, consuming 99% of server resources for the hour or so needed to pull the content, regular site users be damned. A configuration like this is reasonably low maintenance while limiting the damage by bots.