4 ms·
So.. Does Google obey these TOS? And - are they obliged to?
by kiallmacinnes 14y ago
So.. Does Google obey these TOS? And - are they obliged to?
- deleted 14y ago[deleted]
- Roedou 14y agoRobots.txt is only an indication of which pages should/shouldn't be visited. Instructions about which pages can be included in a search engine's index (and hence linked to from results pages) would go in a meta tag on the relevant page. Very large sites would love to see a 'noindex' parameter in robots.txt; sadly it's not there.
- gioele 14y ago> Very large sites would love to see a 'noindex' parameter in robots.txt; sadly it's not there. Isn't User-agent: * Disallow: / enough?
- eli 14y agoI don't see how Google could possibly be expected to parse legalese English sentences.
- jvm 14y agoLooks like they are implicitly permitting google via robots.txt: http://www.victoriassecret.com/robots.txt http://www.victoriassecret.com/robots.txt User-agent: ia_archiver Disallow: / User-agent: * Disallow: /commerce/application?namespace=shoppingBag&origin=myMain.jsp&event=link.viewBag Disallow: /commerce/logoff.vs Disallow: /commerce/signin.vs Disallow: /commerce/registration.vs Disallow: /commerce/logon.vs Disallow: /commerce/addressbook/index.jsp Disallow: /commerce/changePasswordJsf.vs Sitemap: http://www.victoriassecret.com/sitemap.xml