4 ms·
Is there a technical reason that other crawlers can't just follow the googlebot rules? It's great that bots seem to obey the wishes of site owners, but I'm genu
by podgib 11y ago
Is there a technical reason that other crawlers can't just follow the googlebot rules? It's great that bots seem to obey the wishes of site owners, but I'm genuinely surprised that the yandex bot doesn't just follow the Google bot rule and ignore the general disallow rule.