3 ms·
might as well do the full list: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt https://github.com/ai-robots-txt/ai.robots.txt/blob/main/ro
by nikcub 2y ago
might as well do the full list:
https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob...
cloudflare have a button for this:
https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo...
- username223 2y agoDon't stop at robots.txt blocking. Look through your access logs, and you'll likely find a few IPs generating a huge amount of traffic. Look them up via "whois," then block the entire IP range if it seems like a bot host. There's no reason for cloud providers to browse my personal site, so if they host crawlers, they get blocked.
- 7373737373 2y agoThank you very much for mentioning these! These parasites deserve to burn in hell for their greed and violation of consent