3 ms·
Facebook's robots.txt looks pretty restrictive: http://facebook.com/robots.txt http://facebook.com/robots.txt Their sitemap doesn't even load for me. (I don'
by jrockway 14y ago
Facebook's robots.txt looks pretty restrictive:
http://facebook.com/robots.txt http://facebook.com/robots.txt
Their sitemap doesn't even load for me.
(I don't use Facebook so I don't know how much this covers, but it's an interesting read compared to http://plus.google.com/robots.txt. http://plus.google.com/robots.txt.)
- nwh 14y agoUser-agent: * Disallow: / Yep, pretty much everything.