3 ms·
Curious if you have any citations for this remark. I have always been under the impression that robots.txt is entirely optional, both to implement and to respec
by uxp 16y ago
Curious if you have any citations for this remark. I have always been under the impression that robots.txt is entirely optional, both to implement and to respect.
- moultano 16y agoIANAL and looking for other sources, but it's discussed along with many other factors here: http://www.benedict.com/Digital/Internet/Field/Field.aspx http://www.benedict.com/Digital/Internet/Field/Field.aspx Essentially, the existence of robots.txt and meta noindex has made courts more comfortable ruling that the index of search engines constitutes fair use.
- uxp 16y agoBut this still doesn't specifically answer the question, "What is the penalty for ignoring 'User-agent:* Disallow: /' in robots.txt, or meta tag no-index", only that without a robots.txt or no-index tag, crawlers can't be found liable for accessing information that is freely available to them. Interesting link, however. I was unaware of that case.