4 ms·

>> either ignore robots.txt or skip crawling your site. That’s not really how it works now. Google pretty much ignores robots.txt now and crawls your site anyw

>> either ignore robots.txt or skip crawling your site. That’s not really how it works now. Google pretty much ignores robots.txt now and crawls your site anyway.

marginalia_nu 5y ago
Google does obey robots.txt, but there are a lot of crawlers that identify as GoogleBot that are not from google.
- zinekeller 5y ago
  Some web firewalls now even detect fake crawlers by using instructions published by search engines (Google: https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/..., Microsoft: https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26 https://www.bing.com/webmasters/help/how-to-verify-bingbot-3..., Yandex: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html https://yandex.com/support/webmaster/robot-workings/check-ya..., Apple: restrict to 17.0.0.0/8 (their IP range, https://support.apple.com/en-us/HT204683 https://support.apple.com/en-us/HT204683))
- Dma54rhs 5y ago
  It does not, they obey it regarding indexing pages but not crawling. They can and will crawl even if you say no, they've officially said so as well in their docs.
  - dwild 5y ago
    Can you point to your source? From a quick glance on their documentation, it seems like you got it backward. > A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page. https://developers.google.com/search/docs/advanced/robots/intro https://developers.google.com/search/docs/advanced/robots/in... They won't crawl it, but it they already did crawl it or if there's a link to it, it will still be part of their index (thus search results). They even go further on their noindex page, by saying that it need to be crawlable (thus not part of the robots.txt) so that they can see the no index directive on the page... > Important: For the noindex directive to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex directive, and the page can still appear in search results, for example if other pages link to it. https://developers.google.com/search/docs/advanced/crawling/block-indexing https://developers.google.com/search/docs/advanced/crawling/...

>> either ignore robots.txt or skip crawling your site. That’s not really how it works now. Google pretty much ignores robots.txt now and crawls your site anyway.

marginalia_nu 5y ago

Google does obey robots.txt, but there are a lot of crawlers that identify as GoogleBot that are not from google.

zinekeller 5y ago
Some web firewalls now even detect fake crawlers by using instructions published by search engines (Google: https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/..., Microsoft: https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26 https://www.bing.com/webmasters/help/how-to-verify-bingbot-3..., Yandex: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html https://yandex.com/support/webmaster/robot-workings/check-ya..., Apple: restrict to 17.0.0.0/8 (their IP range, https://support.apple.com/en-us/HT204683 https://support.apple.com/en-us/HT204683))
Dma54rhs 5y ago
It does not, they obey it regarding indexing pages but not crawling. They can and will crawl even if you say no, they've officially said so as well in their docs.
- dwild 5y ago
  Can you point to your source? From a quick glance on their documentation, it seems like you got it backward. > A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page. https://developers.google.com/search/docs/advanced/robots/intro https://developers.google.com/search/docs/advanced/robots/in... They won't crawl it, but it they already did crawl it or if there's a link to it, it will still be part of their index (thus search results). They even go further on their noindex page, by saying that it need to be crawlable (thus not part of the robots.txt) so that they can see the no index directive on the page... > Important: For the noindex directive to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex directive, and the page can still appear in search results, for example if other pages link to it. https://developers.google.com/search/docs/advanced/crawling/block-indexing https://developers.google.com/search/docs/advanced/crawling/...