4 ms·

If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. Is that pretty much normal?

• kohtatsu 6y ago
  It's a courtesy, afaict.
  • moonchild 6y ago
    Rather the opposite. Many robots.txt will disallow * but allow googlebot. This allows them to crawl those sites without getting flak for disregarding robots.txt completely.