3 ms·
If you do this, create the robots.txt first, then wait a week or two! Only then activate the actual blackhole. The reason is that robots do not download robot
by ars 16y ago
If you do this, create the robots.txt first, then wait a week or two!
Only then activate the actual blackhole.
The reason is that robots do not download robots.txt each time, they can, and do, cache it for quite a while, especially for sites that don't change much.
- deleted 16y ago[deleted]