5 ms·
For most crawlers, robots.txt will work. For people actually trying, nothing but an IP block and vigilance will help.
by debacle 9y ago
For most crawlers, robots.txt will work. For people actually trying, nothing but an IP block and vigilance will help.