I assume you are concerned about crawlers that do not respect the robots.txt file (which is the polite way to restrict them from indexing your side, but does no

I assume you are concerned about crawlers that do not respect the robots.txt file (which is the polite way to restrict them from indexing your side, but does not provide any actual protection if crawlers chose to ignore the file). Cloudflare has a tool for doing this (now part of their core service): https://blog.cloudflare.com/introducing-scrapeshield-discover-defend-dete/ https://blog.cloudflare.com/introducing-scrapeshield-discove... There's a nice Github repo with some advice on blocking scrapers: https://github.com/JonasCz/How-To-Prevent-Scraping https://github.com/JonasCz/How-To-Prevent-Scraping Finally, you could use a plugin in your Webserver to display a CAPTCHA to visitors from IP addresses that cause a lot of requests to your site. There are many more strategies available (up to creating fake websites / content to lead crawlers astray), but the CAPTCHA solution is the most robust one. It will not be able to protect you against crawlers that use a large source IP pool to access your site though.