So in addition to updating the robots.txt file, which really only blocks a small number of them. Seems CF has been gathering data and profiling these malicious

So in addition to updating the robots.txt file, which really only blocks a small number of them. Seems CF has been gathering data and profiling these malicious agents. This post by CF elaborates a bit further: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo... Basically becomes a game of cat and mouse.