4 ms·
Totally agree, that's conceptually the same problem as robots.txt. As stated in https://www.robotstxt.org/faq/blockjustbad.html https://www.robotstxt.org/faq/bl
by az09mugen 1y ago
Totally agree, that's conceptually the same problem as robots.txt. As stated in https://www.robotstxt.org/faq/blockjustbad.html https://www.robotstxt.org/faq/blockjustbad.html :
> But almost all bad robots ignore /robots.txt, making that pointless.