4 ms·
Do you use this approach? If so, are you willing to share an example robots.txt file? What bots would be allowed? Google, Bing, ...? Those?
by JackGreyhat 3y ago
Do you use this approach?
If so, are you willing to share an example robots.txt file? What bots would be allowed? Google, Bing, ...? Those?
- nuc1e0n 3y agorobots.txt is a standard and has been for many years. It seems that https://www.robotstxt.org https://www.robotstxt.org is the main repository for its definition. Any robots.txt I might suggest wouldn't necessarily be appropriate for anyone else. It's a policy that site stakeholders need to decide for themselves I'd say.