4 ms·
The bad actors don't follow the robots.txt, or rather are bad actors because they don't follow robots.txt neither the informal courtesy practices. Then the main
by undebuggable 5y ago
The bad actors don't follow the robots.txt, or rather are bad actors because they don't follow robots.txt neither the informal courtesy practices. Then the main role of robots.txt is public shaming?
- chrisweekly 5y agoThere are robots which follow robots.txt but which site operators prefer to block or content-restrict, for various reasons. See eg the annotated wikipedia robots.txt me tioned upthread.
- Beltalowda 5y agoThere are really three types of crawlers: good actors, bad actors, and "grey actors". The grey actors can be just by making a mistake (or outright incompetence), or they're testing where the limits are, etc. robots.txt does nothing for the bad actors, but for grey actors it's a bit more complicated as they can move from "grey" to "good". Adding something for that in robots.txt in addition to some other limits (if needed) can be useful.
- flavius29663 5y agoIf you disallow something and it still get crawled, doesn't that reveal them as bad actors?
- Beltalowda 5y agoNot necessarily; could be a bug or other mistake such as incomplete support, or just plain ignorance/incompetence.