4 ms·Google indexers respect robots.txt, so there goes the irony.by avip 8y agoGoogle indexers respect robots.txt, so there goes the irony.pbhjpbhj 8y agoI've heard that they [sometimes] visit but don't index, is that true?p49k 8y agoThey also provide something of value to the operators of the websites they scrape, namely search traffic.