this reminds: webcrawlers are supposed to include an info url with their user agent header ex: Googlebot/2.1 (+http://www.google.com/bot.html http://www.google

this reminds: webcrawlers are supposed to include an info url with their user agent header ex: Googlebot/2.1 (+http://www.google.com/bot.html http://www.google.com/bot.html)