3 ms·
Google boutiques.com has no robots.txt
- yread 16y agoCould it be user agent filtering?
- ithkuil 16y agoWell, you can try it with: wget -O- http://www.boutiques.com/robots.txt http://www.boutiques.com/robots.txt -U "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) http://www.google.com/bot.html) and you get the same error. I tried a couple of other bot user agent strings from http://www.useragentstring.com/ http://www.useragentstring.com/, same result
- mooism2 16y agoSo?...
- AndrewDucker 16y agoWhy would it?
- andrenotgiant 16y agothey have NOINDEX, NOFOLLOW robots META tags that tell search engine crawlers to go away. Maybe it's their way of playing nice with all of the merchants they are listing.