3 ms·
robots.txt. If all requests sent by robots would clearly identify themselves, the server would easily block all of them. But if they fake their user agent to l
by ikeboy 9y ago
robots.txt.
If all requests sent by robots would clearly identify themselves, the server would easily block all of them. But if they fake their user agent to look like a browser and ignore robots.txt, that's not a good faith request and they shouldn't be able to plead ignorance.
- cmiles74 9y agoI don't believe there's a law requiring the honoring of the robots.txt file. People and services honor the file out of a sense of good manners, not a legal requirement.
- ikeboy 9y agoIt doesn't have to be a specific law. It is a rebuttal to a claim of "I had no idea I shouldn't have requested millions of pages from that site". If you scrape a site that prohibits it in robots.txt, that should be considered notice that they don't want that, for whatever relevant law. (I don't know if this argument would hold up in court, IANAL.)
- cmiles74 9y agoI think I see what you're saying, but I disagree that the robots.txt file should have any legal ramifications. Web site operators have many tools that they can use to limit traffic or protect data and they should make good use of those tools. LinkedIn wants to make their data available publicly, except under certain conditions. In my opinion, if they can't find a technical solution, they should stop making the data available publicly.
- jimktrains2 9y agoWhat is a robot? Why is the User Agent even important? It's not a standardized value. I could send "User-Agent: ikeboy" and it's perfectly valid.