2 ms·
If I remember correctly, it doesn't. The crawler ignores it. The problem is, some people still want to be indexed by the Live crawler, but without the spam in
by Jem 18y ago
If I remember correctly, it doesn't. The crawler ignores it.
The problem is, some people still want to be indexed by the Live crawler, but without the spam in the referrals. At least by blocking the crawler purely in the stats, they still have a chance at ranking with Live without the pain of sorting through hundreds of fake referrals.