3 ms·
And assume you have User-agent: meta-externalagent Disallow: /
by reconnecting 4mo ago
And assume you have
User-agent: meta-externalagent
Disallow: /
- kev009 4mo agoThey don't obey *, they don't get their own entry. I'd rather just poison their data, it's a well known behavior from them. https://www.reddit.com/r/webdev/comments/1sdzd1q/metas_ai_crawler_scraped_my_site_79_million_times/ https://www.reddit.com/r/webdev/comments/1sdzd1q/metas_ai_cr...
- Symbiote 4mo agoI have observed the same from Meta's crawler. User-agent: * Disallow: / on e.g. our preproduction site, Meta is the only big-tech crawler that accesses it, at least with an honest user agent. (Meta also accesses disallowed paths on the production site.)
- reconnecting 4mo agoI'm not defending meta here, but I should mention that meta also uses crawlers to visit pages when someone send a link through their services. User-agent: * can be ignored by bots, but if they ignore the disallow rule for their own UA, they can easily be blocked by network AS.