3 ms·

What did the robots.txt file say? Was that violated? Is robots.txt a legal enough statement? Interesting case.

jat850 16y ago
As robots.txt is entirely optional (there's not even an RFC governing it), what it says or doesn't say isn't even technically violated in any way here, let alone legally. (I apologize, I don't know how to do proper citations on here but http://en.wikipedia.org/wiki/Robots.txt http://en.wikipedia.org/wiki/Robots.txt provides more details.)
- pierrefar 16y ago
  But it is the convention that is widely accepted.
  - jasonlotito 16y ago
    But not the only thing to consider. The article does mention violations of agreements, and not just the scraping. The argument isn't about scraping alone. I could see their being a contract that limited what the user accounts could be used for, and the possibility that the user accounts were limited to particular individuals and what not, as well as the limitation of their use. Of course, all of this is speculation. The point is, it's not JUST a scraping case.
  - jat850 16y ago
    Yep, most services or search engines choose to respect the convention, but if the article indicates properly, it doesn't sound like the robot in question here was trying to be particularly kind or gentle in its work.
wwortiz 16y ago
It seems that it shouldn't matter if a robots.txt was present or not because logins were required to "scrape" the information and that implies that those accounts that had to login probably had to accept some sort of agreement which probably forbid this in some way.
flog 16y ago
The more interesting question is does a site's individual policies apply? Does scraping a site constitute acceptance of the site's terms if no compliance is sought before it's use? Surely Google would be violating millions of such policies. In this case it sounds like a login was required to access the data, in which case the account holder has probably had to agree to such terms. As someone starting a business which plans to use web-crawling to a large extent I find the publicly-accessible side of this very interesting. Unfortunately it's still a very gray area - nobody seems to have an answer for me. The advice seems to be "try it, but you may have to defend it", which means a lot more money than most startups have.
qeorge 16y ago
Interesting question. Anecdotally, I believe Facebook recently took the position that robots.txt has no legal standing: "Their contention was robots.txt had no legal force and they could sue anyone for accessing their site even if they scrupulously obeyed the instructions it contained. The only legal way to access any web site with a crawler was to obtain prior written permission." http://petewarden.typepad.com/searchbrowser/2010/04/how-i-got-sued-by-facebook.html http://petewarden.typepad.com/searchbrowser/2010/04/how-i-go...

As robots.txt is entirely optional (there's not even an RFC governing it), what it says or doesn't say isn't even technically violated in any way here, let alone legally. (I apologize, I don't know how to do proper citations on here but http://en.wikipedia.org/wiki/Robots.txt http://en.wikipedia.org/wiki/Robots.txt provides more details.)

pierrefar 16y ago
But it is the convention that is widely accepted.
- jasonlotito 16y ago
  But not the only thing to consider. The article does mention violations of agreements, and not just the scraping. The argument isn't about scraping alone. I could see their being a contract that limited what the user accounts could be used for, and the possibility that the user accounts were limited to particular individuals and what not, as well as the limitation of their use. Of course, all of this is speculation. The point is, it's not JUST a scraping case.
- jat850 16y ago
  Yep, most services or search engines choose to respect the convention, but if the article indicates properly, it doesn't sound like the robot in question here was trying to be particularly kind or gentle in its work.

wwortiz 16y ago

It seems that it shouldn't matter if a robots.txt was present or not because logins were required to "scrape" the information and that implies that those accounts that had to login probably had to accept some sort of agreement which probably forbid this in some way.

flog 16y ago

The more interesting question is does a site's individual policies apply? Does scraping a site constitute acceptance of the site's terms if no compliance is sought before it's use? Surely Google would be violating millions of such policies. In this case it sounds like a login was required to access the data, in which case the account holder has probably had to agree to such terms. As someone starting a business which plans to use web-crawling to a large extent I find the publicly-accessible side of this very interesting. Unfortunately it's still a very gray area - nobody seems to have an answer for me. The advice seems to be "try it, but you may have to defend it", which means a lot more money than most startups have.

qeorge 16y ago

Interesting question. Anecdotally, I believe Facebook recently took the position that robots.txt has no legal standing: "Their contention was robots.txt had no legal force and they could sue anyone for accessing their site even if they scrupulously obeyed the instructions it contained. The only legal way to access any web site with a crawler was to obtain prior written permission." http://petewarden.typepad.com/searchbrowser/2010/04/how-i-got-sued-by-facebook.html http://petewarden.typepad.com/searchbrowser/2010/04/how-i-go...