4 ms·
robots.txt is not a security feature. It's a courtesy that saves resources and makes life a lot easier for everyone who is willing to play by the rules.
by Goladus 19y ago
robots.txt is not a security feature. It's a courtesy that saves resources and makes life a lot easier for everyone who is willing to play by the rules.