2 ms·
They are more accessible here on HN, though, because crawlers that obey robots.txt will not see it on git.savannah.gnu.org but will see it here on HN.
by tzs 10y ago
They are more accessible here on HN, though, because crawlers that obey robots.txt will not see it on git.savannah.gnu.org but will see it here on HN.
- yellowapple 10y agoI suspect that if a crawler is being used to farm email addresses for spamming, it's highly unlikely that robots.txt would be any deterrent whatsoever.
- mysidia1 10y agoSpammers' crawlers use URLs obtained from search engines and public sources. If the whole directory is blocked in robots.txt, it WILL reduce crawling activity massively.