5 ms·
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Disallow: / User-Agent: googlebot Disallow: /search Al
by forgotusername 14y ago
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /
User-Agent: googlebot
Disallow: /search
Allow: /
Woah, that is surprising. I note Bing has blogspot in its index anyway. Perhaps they use the ATOM API when they see a Blogspot URL? (technically not 'crawling')
- abraham 14y agoWhere are you getting that? It doesn't match what I'm seeing. http://googleblog.blogspot.com/robots.txt http://googleblog.blogspot.com/robots.txt
- forgotusername 14y agoGoogled for "blogspot", picked first random domain I saw, "weliveyoung.blogspot.com", fetched "weliveyoung.blogspot.com/robots.txt" with curl, got a redirect to "weliveyoung.blogspot.co.uk/robots.txt", fetched that, voila. Perhaps there is a user setting that controls it.
- abraham 14y agoLooks like you can use whatever you want. The one I linked to is the default. https://support.google.com/blogger/bin/answer.py?hl=en&answer=2472627 https://support.google.com/blogger/bin/answer.py?hl=en&a...