3 ms·

So they still can use up my resources?

• Google234 5y ago
  How can they know without looking? Again, you could also use the webmasters tools to block it...
  • uglygoblin 5y ago
    Robots.txt can accomplish the same thing which means they periodically grab a single text file from you.
    • Google234 5y ago
      How dare they!
    • account42 5y ago
      > robots.txt can accomplish the same thing Not exactly - robots.txt will prevent Google from crawling the page but it might still show up in search results (without a snippet or your page title) based on how others link to it.