4 ms·
Fair pushback. A few angles: 1. The premise is partly wrong. Major commercial agents -- the ones with legal teams and reputational risk -- already follow robot
by jaspervanveen 8mo ago
Fair pushback. A few angles:
1. The premise is partly wrong. Major commercial agents -- the ones with legal teams and reputational risk -- already follow robots.txt. Google, Bing, all the big crawlers comply. The ones that don't were always going to ignore any standard. You design standards for good actors, not bad ones.
2. agents.txt has a positive use case robots.txt doesn't. If a site declares "here's my MCP server, use that instead of scraping my HTML" -- a well-built agent actively wants to use it. It's faster, more reliable, and lower risk than parsing HTML. Compliance here is self-interested, not just ethical.
3. Non-compliance becomes legally actionable. Once there's a published standard, ignoring it strengthens ToS violation and CFAA claims. "You had a machine-readable policy and the agent ignored it" is a much cleaner legal argument than it is today.
4. The same argument was made about robots.txt in 1994. It's still here 30 years later because the ecosystem of good actors is larger than the ecosystem of bad ones -- and it keeps growing.
The goal isn't to stop bad actors. It's to give good actors a common language.