4 ms·
Actually I do work on a website that we do not want crawled. It would be handy to know which things to leave out to make it less likely your site will be crawle
by irrational 9y ago
Actually I do work on a website that we do not want crawled. It would be handy to know which things to leave out to make it less likely your site will be crawled or show up high in a google result.
- mileycyrusXOXO 9y agoJust define a robot.txt file instructing the crawler to not index your website. See: http://www.robotstxt.org/ http://www.robotstxt.org/
- seanwilson 9y agoThis is a common misconception: https://support.google.com/webmasters/answer/7424835?hl=en#h17 https://support.google.com/webmasters/answer/7424835?hl=en#h... > robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. If it's something like a staging site, you'd be better putting it behind a login screen to prevent mistakes anyway.