Awesome post, from a fellow Brazilian! We did a very similar implementation (although not distributed) for a similar problem, using Redis and Laravel. We had

Awesome post, from a fellow Brazilian! We did a very similar implementation (although not distributed) for a similar problem, using Redis and Laravel. We had MANY people crawling our website, and we would prefer that they use our API for that. Using Redis we block IPs who accessed our website more than X times not logged-in (200 URLs right now). We also had the requirement that all good bots(Bing, Baidu, Google) should pass-thru without blocks or any slowdown. Another requirement, was that those good bots should be verified(Reverse & Forward DNS Lookup) before entering out good bot list. It is working great for our high-traffic website ( 2 Mi hits/day). You can check our work here: https://github.com/Potelo/laravel-block-bots https://github.com/Potelo/laravel-block-bots