4 ms·
I don't believe there's a law requiring the honoring of the robots.txt file. People and services honor the file out of a sense of good manners, not a legal requ
by cmiles74 9y ago
I don't believe there's a law requiring the honoring of the robots.txt file. People and services honor the file out of a sense of good manners, not a legal requirement.
- ikeboy 9y agoIt doesn't have to be a specific law. It is a rebuttal to a claim of "I had no idea I shouldn't have requested millions of pages from that site". If you scrape a site that prohibits it in robots.txt, that should be considered notice that they don't want that, for whatever relevant law. (I don't know if this argument would hold up in court, IANAL.)
- cmiles74 9y agoI think I see what you're saying, but I disagree that the robots.txt file should have any legal ramifications. Web site operators have many tools that they can use to limit traffic or protect data and they should make good use of those tools. LinkedIn wants to make their data available publicly, except under certain conditions. In my opinion, if they can't find a technical solution, they should stop making the data available publicly.