3 ms·

This has me pondering whether there should there be some kind of robots.txt check involved. There are implications of arbitrarily requesting data on someone els

• chime 16y ago
  Absolutely. If it's automated and can be used on many different pages (basically a bot script), it should respect robots.txt.
• viraptor 16y ago
  Has TOS-like set of rules been ever proven valid / enforceable? It seems to me like they are just a wish list / guidelines for not registered users. It's a different thing when you actually make users accept those terms during registration of course - then both parties know the rules. But for anonymous visitor? The service provides the content without / before showing the rules. IMHO that means any "don't use robots" rules are not enforceable at all. I don't mean to imply that robots.txt should not be respected though (it should if you're a whitehat) - just that it's legal to disregard it if you don't explicitly accept the terms.
  • gojomo 16y ago
    I don't know if it's yet been tested in court, but the robots.txt conventions have over a decade of widespread custom behind them. It would be hard for anyone competent enough to do sophisticated crawling to plead ignorance of them. Custom counts in law: http://en.wikipedia.org/wiki/Custom_%28law%29 http://en.wikipedia.org/wiki/Custom_%28law%29