3 ms·

robots.txt won't work?

• Jem 18y ago
  If I remember correctly, it doesn't. The crawler ignores it. The problem is, some people still want to be indexed by the Live crawler, but without the spam in the referrals. At least by blocking the crawler purely in the stats, they still have a chance at ranking with Live without the pain of sorting through hundreds of fake referrals.
• eli 18y ago
  Not only does it not work, but at least in my case the bot was hammering the site with many requests per seconed. I blocked it with ModSecurity over a year ago.