2 ms·
Using the googlebot User-Agent string gives me: User-agent: * Disallow: This robots.txt should allow all bots to search the entire website. However, I thi
by cpeterso 14y ago
Using the googlebot User-Agent string gives me:
User-agent: *
Disallow:
This robots.txt should allow all bots to search the entire website. However, I think Google also penalizes websites that serve different content to googlebot than to non-bot user agents.