4 ms·
It's a courtesy, afaict.
by kohtatsu 6y ago
It's a courtesy, afaict.
- moonchild 6y agoRather the opposite. Many robots.txt will disallow * but allow googlebot. This allows them to crawl those sites without getting flak for disregarding robots.txt completely.