4 ms·> To be fair, Google respects robots.txt Not all of their crawlers respect robots.txtby zippy786 11y ago> To be fair, Google respects robots.txt Not all of their crawlers respect robots.txtcbr 11y agoDo you have an example? A crawler ignoring robots.txt sounds like a bug, and if you have details I can file a bug with the appropriate team.zippy786 11y agoFeedFetcher User Agent