3 ms·
I run a cronjob every 5 minutes that parses httpd access log. If there's an IP with abnormally large request number, it blocks it. Most crawlers will make hund
by z92 9y ago
I run a cronjob every 5 minutes that parses httpd access log. If there's an IP with abnormally large request number, it blocks it.
Most crawlers will make hundreds of requests in five minutes, while legitimate viewers will make be bellow 100.
- inglor 9y agoIf an attacker uses something like Illuminati (or just abuses Tor or another botnet) then they can (and should) distribute HTTP requests across different IPs. This is pretty cheap to do and I've seen it done before in several places I've worked (on the defending side).
- lazyjones 9y agoThis is ineffective and dangerous where ISPs allow switching dynamic IP addresses with no delays. Large german ISPs do this, so your abusive scraper will just continue with a new IP address while some legitimate user, who is unlucky enough to get the abuser's old IP address, is blocked.