3 ms·
robots.txt in the past has been the legal defense against claims of copyright infringement.
by moultano 16y ago
robots.txt in the past has been the legal defense against claims of copyright infringement.
- uxp 16y agoCurious if you have any citations for this remark. I have always been under the impression that robots.txt is entirely optional, both to implement and to respect.
- moultano 16y agoIANAL and looking for other sources, but it's discussed along with many other factors here: http://www.benedict.com/Digital/Internet/Field/Field.aspx http://www.benedict.com/Digital/Internet/Field/Field.aspx Essentially, the existence of robots.txt and meta noindex has made courts more comfortable ruling that the index of search engines constitutes fair use.
- uxp 16y agoBut this still doesn't specifically answer the question, "What is the penalty for ignoring 'User-agent:* Disallow: /' in robots.txt, or meta tag no-index", only that without a robots.txt or no-index tag, crawlers can't be found liable for accessing information that is freely available to them. Interesting link, however. I was unaware of that case.