3 ms·

It's public right now. It won't be once Yahoo pulls the plug on Delicious.

• Tichy 16y ago
  User-agent: * Disallow: / I don't remember the robots.txt rules for sure, but doesn't that mean they don't allow crawlers at all?
  • ivank 16y ago
    That's the rule for crawlers that aren't Slurp, Googlebot, Teoma, or msnbot.
    • Tichy 16y ago
      I noticed the extra rules, but I am neither Slurp, Googlebot, Teoma nor msnbot :-(
  • mfukar 16y ago
    robots.txt is merely a suggestion.