4 ms·

Robots.txt can accomplish the same thing which means they periodically grab a single text file from you.

• Google234 5y ago
  How dare they!
• account42 5y ago
  > robots.txt can accomplish the same thing Not exactly - robots.txt will prevent Google from crawling the page but it might still show up in search results (without a snippet or your page title) based on how others link to it.