4 ms·
robots.txt is a standard and has been for many years. It seems that https://www.robotstxt.org https://www.robotstxt.org is the main repository for its definitio
by nuc1e0n 3y ago
robots.txt is a standard and has been for many years. It seems that https://www.robotstxt.org https://www.robotstxt.org is the main repository for its definition.
Any robots.txt I might suggest wouldn't necessarily be appropriate for anyone else. It's a policy that site stakeholders need to decide for themselves I'd say.