4 ms·

My book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and t

by bwb 1y ago

My book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and the bots are getting smarter. I wish there were a better way to solve this.

skydhash 1y ago
If you're not updating the publicly accessible part of the database open, try to see if you can put some cache strategy up and let cloudflare take the hit.
- bwb 1y ago
  Yep, all but one page type is heavily cached at multiple levels. We are working to get the rest and improve it further... just annoying as they don't even respect limits..
weaksauce 1y ago
put a honeypot link in your site that only robots will hit because it’s hidden. make sure it’s not in robots.txt or ban it if you can in robots.txt. setup a rule that any ip that hits that link will get a 1 day ban in your fail2ban or the like.
- bwb 1y ago
  Got a good link to something on github that does this? I have to make sure legit bots don't get hit, as a huge percent of our traffic which helps the project stay active is from google, etc.
  - subscribed 1y ago
    I did it manually and got fail2ban to read the access log anyway. The it's the permanent iptables rule, but could be CF API call as well.
  - weaksauce 1y ago
    here's one example using rack attack on a rails app: https://github.com/pinballmap/pbm/blob/302ac638850711878ac618847b323fa7c2d9e3e3/config/initializers/rack_attack.rb#L27 https://github.com/pinballmap/pbm/blob/302ac638850711878ac61... https://github.com/pinballmap/pbm/blob/302ac638850711878ac618847b323fa7c2d9e3e3/config/robots.production.txt#L63 https://github.com/pinballmap/pbm/blob/302ac638850711878ac61... but it only bans for 3 hours. if they don't respect the hidden link and robots.txt they get banned.
p3rls 1y ago
At this point I'd take a thermostat that can read when my dashboard starts getting heated (always the same culprits causing these same server spikes) and flicks attack mode on for cloudflare.... it's so ridiculous trying to run anything that's not a wordpress these days
shepherdjerred 1y ago
ah, you're the one who stopped me from being jerred@shepherd.com!
- bwb 1y ago
  hah eh?

by bwb 1y ago

skydhash 1y ago

If you're not updating the publicly accessible part of the database open, try to see if you can put some cache strategy up and let cloudflare take the hit.

bwb 1y ago
Yep, all but one page type is heavily cached at multiple levels. We are working to get the rest and improve it further... just annoying as they don't even respect limits..

weaksauce 1y ago

put a honeypot link in your site that only robots will hit because it’s hidden. make sure it’s not in robots.txt or ban it if you can in robots.txt. setup a rule that any ip that hits that link will get a 1 day ban in your fail2ban or the like.

bwb 1y ago
Got a good link to something on github that does this? I have to make sure legit bots don't get hit, as a huge percent of our traffic which helps the project stay active is from google, etc.
- subscribed 1y ago
  I did it manually and got fail2ban to read the access log anyway. The it's the permanent iptables rule, but could be CF API call as well.
- weaksauce 1y ago
  here's one example using rack attack on a rails app: https://github.com/pinballmap/pbm/blob/302ac638850711878ac618847b323fa7c2d9e3e3/config/initializers/rack_attack.rb#L27 https://github.com/pinballmap/pbm/blob/302ac638850711878ac61... https://github.com/pinballmap/pbm/blob/302ac638850711878ac618847b323fa7c2d9e3e3/config/robots.production.txt#L63 https://github.com/pinballmap/pbm/blob/302ac638850711878ac61... but it only bans for 3 hours. if they don't respect the hidden link and robots.txt they get banned.

p3rls 1y ago

At this point I'd take a thermostat that can read when my dashboard starts getting heated (always the same culprits causing these same server spikes) and flicks attack mode on for cloudflare.... it's so ridiculous trying to run anything that's not a wordpress these days

shepherdjerred 1y ago

ah, you're the one who stopped me from being jerred@shepherd.com!

bwb 1y ago
hah eh?