3 ms·

User-agent: * Disallow: / I don't remember the robots.txt rules for sure, but doesn't that mean they don't allow crawlers at all?

• ivank 16y ago
  That's the rule for crawlers that aren't Slurp, Googlebot, Teoma, or msnbot.
  • Tichy 16y ago
    I noticed the extra rules, but I am neither Slurp, Googlebot, Teoma nor msnbot :-(
• mfukar 16y ago
  robots.txt is merely a suggestion.