3 ms·
Kinda like robots.txt, there's nothing preventing a crawler from ignoring it or omitting it and just doing whatever it wants. Slapping in a massive header comme
by Adamantcheese 8y ago
Kinda like robots.txt, there's nothing preventing a crawler from ignoring it or omitting it and just doing whatever it wants. Slapping in a massive header comment might work.