3 ms·

You’re giving permission to scrape by way of having a page accessible to the general public. If googlebot is bothering you so much, install a two line robots.tx

You’re giving permission to scrape by way of having a page accessible to the general public. If googlebot is bothering you so much, install a two line robots.txt and it’ll go away.

jiggunjer 9y ago
robots can ignore your /robots.txt.
- Karunamon 9y ago
  Googlebot, and every other search engine robot of note. Putting a page up on the public internet includes the risk that it may be visited.