2 ms·
I thought robots.txt was provided an opt out for all web crawlers, not just vendor specific ones? What would be the use case for not using "*" if you didn't wan
by seanmcdirmid 2y ago
I thought robots.txt was provided an opt out for all web crawlers, not just vendor specific ones? What would be the use case for not using "*" if you didn't want something to be crawled?
- atty 2y agoIt’s very reasonable to want Google and Bing to index your page for search, but not have your data collected for training models, i think. I’m not familiar with robots.txt to know if it has a whitelisting mechanism
- seanmcdirmid 2y agoI see a lot of Robots.txt files that use non-wildcards for Allow but what would be the use case for using a non-widlcard for Disallow?
- simonw 2y agorobots.txt has always had mechanisms for allowing or denying specific crawlers. Most of the AI labs that crawl the web support something like this now, here are a few relevant examples from the NY Times robots.txt file for example: https://www.nytimes.com/robots.txt https://www.nytimes.com/robots.txt User-agent: anthropic-ai Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: FacebookBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: /
- indigo0086 2y agoIf apple hired one billion employees to look at websites and manually write the data they find, they wound naturally not respect robots.txt.
- gmuslera 2y agoNot exactly. It opt outs of respectful web crawlers by any name, or if you specify it, by some name in particular. Misbehaved ones requires a bit of more effort.