3 ms·
... which rule would be obeyed only by legitimate, robots.txt-honoring crawlers. This reminds me of the anti-piracy messages shown (solely) to viewers of legall
by chrisweekly 9y ago
... which rule would be obeyed only by legitimate, robots.txt-honoring crawlers. This reminds me of the anti-piracy messages shown (solely) to viewers of legally-purchased media. Similar "logic", similar (counter-productive) "effectiveness".