4 ms·
As robots.txt is entirely optional (there's not even an RFC governing it), what it says or doesn't say isn't even technically violated in any way here, let alon
by jat850 17y ago
As robots.txt is entirely optional (there's not even an RFC governing it), what it says or doesn't say isn't even technically violated in any way here, let alone legally.
(I apologize, I don't know how to do proper citations on here but http://en.wikipedia.org/wiki/Robots.txt http://en.wikipedia.org/wiki/Robots.txt provides more details.)
- pierrefar 17y agoBut it is the convention that is widely accepted.
- jasonlotito 17y agoBut not the only thing to consider. The article does mention violations of agreements, and not just the scraping. The argument isn't about scraping alone. I could see their being a contract that limited what the user accounts could be used for, and the possibility that the user accounts were limited to particular individuals and what not, as well as the limitation of their use. Of course, all of this is speculation. The point is, it's not JUST a scraping case.
- jat850 17y agoYep, most services or search engines choose to respect the convention, but if the article indicates properly, it doesn't sound like the robot in question here was trying to be particularly kind or gentle in its work.