Well, you can try it with: wget -O- http://www.boutiques.com/robots.txt http://www.boutiques.com/robots.txt -U "Mozilla/5.0 (compatible; Googlebot/2.1; +http:/

Well, you can try it with: wget -O- http://www.boutiques.com/robots.txt http://www.boutiques.com/robots.txt -U "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) http://www.google.com/bot.html) and you get the same error. I tried a couple of other bot user agent strings from http://www.useragentstring.com/ http://www.useragentstring.com/, same result