I'm not sure I'd classify most of these as AI agents, it's mostly web crawlers. (Though the definition of agent is fuzzy enough that I suppose you could lump th

I'm not sure I'd classify most of these as AI agents, it's mostly web crawlers. (Though the definition of agent is fuzzy enough that I suppose you could lump them in). I'm also curious, will adding "Common Crawl" to the user-agent disallow list in your robots.txt actually do anything? Added context: I run a tracker of vetted AI agents with verticalized use cases https://staf.ai https://staf.ai