4 ms·
It wouldn't work. The crawlers that cause the problems are not the ones that pay attention to voluntary management systems like robots.txt.
by timr 13y ago
It wouldn't work. The crawlers that cause the problems are not the ones that pay attention to voluntary management systems like robots.txt.