9 ms·

There’s a thing called a robot.txt.

• xondono 6y ago
  Yes, and google has been observed several times to ignore the robots.txt file
  • Google234 6y ago
    No, this is true in general. Maybe there’s been some bugs (I would be surprised, can you link these multiple examples?) but Google respects robot.txt files
• pkilgore 6y ago
  Even assuming Google would not ignore that for a popular enough site (they would know!), there is no robot.txt setting that says "index my site, but please don't steal my content, thanks."
• throwaway189262 6y ago
  Yeah then they won't index you at all and you lose all your traffic instantly. How about a setting to index my site without scaping my content and using it all over the place without attribution