3 ms·
They still scrape and index sites blocked by robots.txt, but they often don’t display those sites in their SERPs (but sometimes they still do)
by jjohansson 7y ago
They still scrape and index sites blocked by robots.txt, but they often don’t display those sites in their SERPs (but sometimes they still do)
- C14L 7y agoNever seen it in any logs in 20 years. Do you have a source for that claim?
- jjohansson 7y agoLooks like I’m wrong. They will rank pages blocked by robots.txt, but won’t index them. https://www.google.ca/amp/s/www.searchenginejournal.com/googles-john-mueller-explains-how-pages-blocked-by-robots-txt-are-ranked/316061/amp/ https://www.google.ca/amp/s/www.searchenginejournal.com/goog...