3 ms·

Yeah, it is. Robots.txt is just a way of politely asking people not to crawl your site. It doesn't have anything to do with the law.

• 4684499 9y ago
  Thank you. I'm not sure if this analogy is correct but isn't that like a host of a museum telling people do not take pictures yet someone did it anyway and sold the pictures he took?
  • dawidloubser 9y ago
    I don't quite think so. There are specific laws governing the use of of, say, photographic equipment within a private property. If a web server, on other other hand, willingly serves content to both a browser being operated by a human, as well as screen-scraping software, then it shouldn't try to prescribe how the screen-scraper uses that information. It would be the equivalent to, every time, asking somebody that works in the museum if you can take a picture, and them saying "yes", and then wanting to complain (or sue) afterwards.
    • icebraining 9y ago
      If a web server, on other other hand, willingly serves content to both a browser being operated by a human, as well as screen-scraping software, then it shouldn't try to prescribe how the screen-scraper uses that information. That's not what was happening here. LinkedIn's server was blocking HiQ, and HiQ sued LinkedIn to prevent them from doing that.