3 ms·
There is a * user agent that can be used to prevent all bots that honor robots.txt from crawling a site. Specific user agents can then be granted additional acc
by nuc1e0n 3y ago
There is a * user agent that can be used to prevent all bots that honor robots.txt from crawling a site. Specific user agents can then be granted additional access afterward.
- JackGreyhat 3y agoDo you use this approach? If so, are you willing to share an example robots.txt file? What bots would be allowed? Google, Bing, ...? Those?
- nuc1e0n 3y agorobots.txt is a standard and has been for many years. It seems that https://www.robotstxt.org https://www.robotstxt.org is the main repository for its definition. Any robots.txt I might suggest wouldn't necessarily be appropriate for anyone else. It's a policy that site stakeholders need to decide for themselves I'd say.