3 ms·
Deny-listing/banning bad crawlers is fine. Especially if they ignore the robots.txt. But allow-listing particular crawlers only is collusion.
by whazor 1y ago
Deny-listing/banning bad crawlers is fine. Especially if they ignore the robots.txt.
But allow-listing particular crawlers only is collusion.