4 ms·

Update 2: After banning close to 2k individual IPs, it looks like I got it under control, for now. I wonder how banning so many IPs affects CloudFlare performa

by santah 6y ago

Update 2: After banning close to 2k individual IPs, it looks like I got it under control, for now. I wonder how banning so many IPs affects CloudFlare performance and if I should optimize it to block whole IP ranges instead ...

aclelland 6y ago
Are the IPs from the same ARN block? Cloudflare should show you that in the Firewall events page. If they're using IPs from a single VPN/Server company then you might be able to just block the ASN. You might also be able to find common user agent headers through the CF firewall page and block them based on their UA. That'd not work if their scraper tool was randomizing the UA string but quite a few of them don't
- santah 6y ago
  Including all IPs I blocked today, they're spread between 5 different ASNs. I may resort to blocking them eventually, but for now - individually blocking the IPs (even in the thousands as it is) - seems to be working well enough. As for user agent - they're using a very common, real browser user agent that's impossible to distinct from legit users.
  - Gys 6y ago
    Do you have many users in Russia? You could block the whole country ;-) Russia has no GDPR or something. So you could put (special key in) a cookie? They probably do not process it so subsequent requests without a cookie are to be discarded?
    - lewiscollard 6y ago
      > Russia has no GDPR or something. So you could put (special key in) a cookie? It is entirely permissible under the GDPR to use cookies for security purposes.
  - eythian 6y ago
    If you can look closer at the HTTP requests, it may still be distinct. For example, the header order may not match any legitimate browser, or some other header doesn't make sense in the context of that UA.
santah 6y ago
Update 3: After about a day - they expanded their pool of IPs to more than 2k new ones. I blocked what I could and then I blocked the whole country (Russia) behind a CloudFlare javasript challenge (so that any legit traffic that wanted to pass it - could). Everything stayed blocked for about a day and now it seems they gave up: - all domains that pulled dynamic content from my site now show some other content and do not try to scrape Next Episode - all static domains that delivered the cached version of the Next Episode homepage now deliver some other website(s) - CloudFlare shows no further traffic on the firewall rules I've set (so the bots from those IPs are, for now, gone) As this post is relatively old now (more than 3 days) I doubt many people will go back and check for new developments, but I wanted to give this the proper closure and let you know of the apparent happy end.
- jochoa 6y ago
  Hey, I read it! Thanks for the update! Now I see why some people block countries or IP ranges in their sites.

by santah 6y ago

aclelland 6y ago

Are the IPs from the same ARN block? Cloudflare should show you that in the Firewall events page. If they're using IPs from a single VPN/Server company then you might be able to just block the ASN. You might also be able to find common user agent headers through the CF firewall page and block them based on their UA. That'd not work if their scraper tool was randomizing the UA string but quite a few of them don't

santah 6y ago
Including all IPs I blocked today, they're spread between 5 different ASNs. I may resort to blocking them eventually, but for now - individually blocking the IPs (even in the thousands as it is) - seems to be working well enough. As for user agent - they're using a very common, real browser user agent that's impossible to distinct from legit users.
- Gys 6y ago
  Do you have many users in Russia? You could block the whole country ;-) Russia has no GDPR or something. So you could put (special key in) a cookie? They probably do not process it so subsequent requests without a cookie are to be discarded?
  - lewiscollard 6y ago
    > Russia has no GDPR or something. So you could put (special key in) a cookie? It is entirely permissible under the GDPR to use cookies for security purposes.
- eythian 6y ago
  If you can look closer at the HTTP requests, it may still be distinct. For example, the header order may not match any legitimate browser, or some other header doesn't make sense in the context of that UA.

santah 6y ago

Update 3: After about a day - they expanded their pool of IPs to more than 2k new ones. I blocked what I could and then I blocked the whole country (Russia) behind a CloudFlare javasript challenge (so that any legit traffic that wanted to pass it - could). Everything stayed blocked for about a day and now it seems they gave up: - all domains that pulled dynamic content from my site now show some other content and do not try to scrape Next Episode - all static domains that delivered the cached version of the Next Episode homepage now deliver some other website(s) - CloudFlare shows no further traffic on the firewall rules I've set (so the bots from those IPs are, for now, gone) As this post is relatively old now (more than 3 days) I doubt many people will go back and check for new developments, but I wanted to give this the proper closure and let you know of the apparent happy end.

jochoa 6y ago
Hey, I read it! Thanks for the update! Now I see why some people block countries or IP ranges in their sites.