3 ms·
It's for controlling search results; blocking overzealous crawlers is a convenient side-effect. robots.txt also blocks indexing of non-html files.
by beoba 16y ago
It's for controlling search results; blocking overzealous crawlers is a convenient side-effect.
robots.txt also blocks indexing of non-html files.