3 ms·

Can someone explain to me why didn't the robots.txt apply to HiQ's crawler? Is crawling still legal while the robots.txt disallows your crawler?

• drngdds 9y ago
  Yeah, it is. Robots.txt is just a way of politely asking people not to crawl your site. It doesn't have anything to do with the law.
  • 4684499 9y ago
    Thank you. I'm not sure if this analogy is correct but isn't that like a host of a museum telling people do not take pictures yet someone did it anyway and sold the pictures he took?
    • dawidloubser 9y ago
      I don't quite think so. There are specific laws governing the use of of, say, photographic equipment within a private property. If a web server, on other other hand, willingly serves content to both a browser being operated by a human, as well as screen-scraping software, then it shouldn't try to prescribe how the screen-scraper uses that information. It would be the equivalent to, every time, asking somebody that works in the museum if you can take a picture, and them saying "yes", and then wanting to complain (or sue) afterwards.
      • icebraining 9y ago
        If a web server, on other other hand, willingly serves content to both a browser being operated by a human, as well as screen-scraping software, then it shouldn't try to prescribe how the screen-scraper uses that information. That's not what was happening here. LinkedIn's server was blocking HiQ, and HiQ sued LinkedIn to prevent them from doing that.