4 ms·

Please don't block everything but Googlebot in robots.txt

• podgib 11y ago
  Is there a technical reason that other crawlers can't just follow the googlebot rules? It's great that bots seem to obey the wishes of site owners, but I'm genuinely surprised that the yandex bot doesn't just follow the Google bot rule and ignore the general disallow rule.
  • Volundr 11y ago
    No technical reason. Following robots.txt is purely self-enforced, crawlers don't even have to read it if they don't want to. It's just part of being a good netizen.
    • podgib 11y ago
      Thanks. That's what I suspected - I'm just surprised that yandex is behaving like a good netizen :)