4 ms·

Websites can choose whether to be scraped or not, and google respects that choice

• hire_charts 6y ago
  It's nice of google to respect the choice to effectively become undiscoverable to anyone searching for your product.
  • ben_w 6y ago
    You sound like you have a third opinion, one which would allow a website to appear on Google in response to user searches without Google knowing anything about what the website contains?
    • DangitBobby 6y ago
      No, they are sarcastically indicating that you don't really have a choice, because if you aren't indexed by Google, practically speaking, you might as well not have a website.
• k4rli 6y ago
  To be fair if being scraped were opt-in (Robots Inclusion Standard?) instead of opt-out, we might have a completely different technological world. Who knows.
• intricatedetail 6y ago
  By creating an account and using their webmaster tools giving up even more of your data?
  • mitchdoogle 6y ago
    No you just have to put a noindex header on requests to your website
    • intricatedetail 6y ago
      So they still can use up my resources?
      • Google234 6y ago
        How can they know without looking? Again, you could also use the webmasters tools to block it...
        • uglygoblin 6y ago
          Robots.txt can accomplish the same thing which means they periodically grab a single text file from you.
          • Google234 6y ago
            How dare they!
          • account42 6y ago
            > robots.txt can accomplish the same thing Not exactly - robots.txt will prevent Google from crawling the page but it might still show up in search results (without a snippet or your page title) based on how others link to it.