6 ms·
I'm still not sure this is going to be very effective, as so many of the worst offenders don't identify themselves as bots, and often change their user agent. H
by cratermoon 1y ago
I'm still not sure this is going to be very effective,
as so many of the worst offenders don't identify themselves as bots,
and often change their user agent.
Has Cloudflare said anything about identifying the bad actors?
- chasd00 1y agoi've mentioned this in a couple replies so maybe i'm wrong but it's up to the client to obey robots.txt. Why would they not just ignore it? Unless there's some legal consequence not complying with robots.txt then why even follow it? There's no technical enforcement of the policies in the file, it's up to the client to honor them.
- kentonv 1y ago> There's no technical enforcement of the policies in the file, it's up to the client to honor them. That's incorrect. Cloudflare does in fact enforce this at a technical level. Cloudflare has been doing bot detection for years and can pretty reliably detect when bots are not following robots.txt and then block them.
- GrayShade 1y agoYes, they have over the years, for example https://blog.cloudflare.com/residential-proxy-bot-detection-using-machine-learning/ https://blog.cloudflare.com/residential-proxy-bot-detection-..., https://blog.cloudflare.com/cloudflare-bot-management-machine-learning-and-more/ https://blog.cloudflare.com/cloudflare-bot-management-machin..., https://blog.cloudflare.com/introducing-bot-analytics/ https://blog.cloudflare.com/introducing-bot-analytics/.
- cratermoon 1y agoBetween those measures, if they are effective and the new blocking, maybe the bigger companies will be induced to behave a little better.