3 ms·
I doubt it. Google's crawling bots (like most search engine bots) follow a published standard for access rules for bots (robots.txt).
by pionar 5y ago
I doubt it. Google's crawling bots (like most search engine bots) follow a published standard for access rules for bots (robots.txt).