4 ms·

Don’t most websites prohibit someone from scraping and harvesting the data on them? Most recently, I can think of Yelp, Amazon, and GitHub prohibiting this, as

Don’t most websites prohibit someone from scraping and harvesting the data on them? Most recently, I can think of Yelp, Amazon, and GitHub prohibiting this, as well as the Aaron Swartz case.

ClumsyPilot 5y ago
"most websites prohibit someone from scraping" This weired mindset where corporations make law shouws up again. The websites have no power to prohibit anything. If they make bytes avaliable, we may do with them as we please so long as its legal.
- Sebb767 5y ago
  They can't throw you in jail over it, but it's within their rights to stop sending you these bytes or kick you off their platform altogether. If they'd try to prevent you from scraping third-party sites it would be making laws; setting up ground rules with their ToS and enforcing them is absolutely fine.
  - dathinab 5y ago
    > but it's within their rights to stop sending you these bytes or kick you off their platform altogether. Actually no. If a platform provides a generally available service they are (in many countries, idk. about the US) not allowed to arbitrary exclude some people they don't like without a legal valid reason. And braking legally not valid/binding terms in a ToS is not a legal valid reason. Just because you write something in your ToS doesn't mean it has any legal relevant meaning, there are limits to what you can put in ToS. And limiting (properly done, privacy respecting) research is often not valid. (Through depends a lot on the country.)
    - throwwebmaster 5y ago
      The FTC is a US government entity.
  - ClumsyPilot 5y ago
    Imagine I am scraping Twitter - maybe I never accepted their TOS and don't even have an account.
    - throwwebmaster 5y ago
      Interesting that you mentioned Twitter. Twitter requires a user account to access content. Try accessing Twitter while logged out.
- throwwebmaster 5y ago
  It’s within anyone’s rights as website maintainers to block malicious IP addresses that scrape or otherwise within their discretion. Nobody is legally forcing websites to allow access to everyone, and accordingly, nobody is altering the law by blocking access to people (crawlers, hackers, spammers, malcontents, or anybody really) that they feel are not welcome. So exercising one’s existing rights isn’t an act of making or altering laws. I suggest reading up on what robots.txt is to further understand this.
  - ClumsyPilot 5y ago
    Hacking and other malicious behavious are actually illegal. Either Crawling does not belong on that list, or google exects should be in jail. Given that crawling is not malicious, what we are discussing now is 'someone is crawling my website in a way I dont like' which is a different gripe. It mighthave some merit, but robots txt is not legally binding.
    - throwwebmaster 5y ago
      As I stated before, Illegal or not, it is within the website owners’ rights to restrict access.
apazzolini 5y ago
Up until June 14th of this year, the ruling was that scraping is legal from the HiQ vs LinkedIn lawsuit[0]. While finding that link for you, I learned that since that date, the Supreme Court vacated the decision back to the lower courts in light of a new decision of theirs.[1] Now I don't think it's clear one way or the other just yet. Any lawyers here with an opinion on how this is going to go? I haven't found any analysis. [0] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn [1] https://www.reuters.com/technology/us-supreme-court-revives-linkedin-bid-shield-personal-data-2021-06-14/ https://www.reuters.com/technology/us-supreme-court-revives-...
- bilbo0s 5y ago
  Well given all this and the consent decree, I'm kind of seeing why FB made this decision. If it's legal, let the FTC say so explicitly. It's probably very bad strategy to allow the privacy leak, and then hope that the FTC agrees with your decision later. No one will be sanctioned for adhering too strictly to consent decrees, but you could be sanctioned for being too loose. So the choice is obvious in that light.
  - throwwebmaster 5y ago
    It will be interesting to see how the FTC will rectify its ongoing fight to hold FB more accountable for protecting its users’ private data with the FTC’s ostensibly contradictory position to allow mass scraping of private user data at scale in this case. Cambridge Analytica was doing roughly the exact same thing, which was precisely what motivated the FTC's involvement in the first place.
- jdminhbg 5y ago
  Scraping is legal, but that doesn't mean the site has to make it easy or possible for you. You can scrape my site, but I can also IP ban scrapers.
kcplate 5y ago
“Prohibit” is an interesting word that in the world of public web publishing of data is actually pretty meaningless. Certainly companies take measures to prevent bulk scraping when they can detect it and have legal remedies if copyrighted or owned material is republished. But simply telling me I am prohibited to hold on to a copy of your site’s data is pretty meaningless when my browser caches your page and you want my browser to cache it to speed your site up.
- throwwebmaster 5y ago
  By accessing the data on a website, you are forming a contract with that website and are subject to the website’s terms and conditions. As a website owner, beyond that, there is no need to tell someone they cannot access your site if you simply block them from accessing your servers instead using a multitude of techniques. Where does caching come into play at all here? You cannot cache content to begin with if the server is blocking access in the first place. And if you have already cached it in the act of violating said website’s terms of service, then you are still not in compliance.
  - nieve 5y ago
    Not according to some rulings the last few years. A contract requires a meeting of minds.
    - throwwebmaster 5y ago
      No. According to many rulings the last few years, continually and systematically accessing a third party’s data under the clear expectation that you are aware of (as well as agreed to) their terms is definitely a meeting of minds.
  - wkavey 5y ago
    I'm pretty sure that first paragraph is false. Until I explicitly agree to a contract, or "terms and conditions", I am not bound by anything. If a site I navigate to embeds content from another website I am not immediately bound by the terms and conditions of that neclsred site, to think otherwise would invite madness. Not to mention the fact that terms and conditions are not contracts. I don't think they carry the same weight, although someone please correct me on this if I am incorrect.
literallyaduck 5y ago
Are these "researchers" going to be charged the same as the martyr Swartz?
- throwwebmaster 5y ago
  The parties here would be the US FTC and FB, so no.

Don’t most websites prohibit someone from scraping and harvesting the data on them? Most recently, I can think of Yelp, Amazon, and GitHub prohibiting this, as well as the Aaron Swartz case.

ClumsyPilot 5y ago

"most websites prohibit someone from scraping" This weired mindset where corporations make law shouws up again. The websites have no power to prohibit anything. If they make bytes avaliable, we may do with them as we please so long as its legal.

Sebb767 5y ago
They can't throw you in jail over it, but it's within their rights to stop sending you these bytes or kick you off their platform altogether. If they'd try to prevent you from scraping third-party sites it would be making laws; setting up ground rules with their ToS and enforcing them is absolutely fine.
- dathinab 5y ago
  > but it's within their rights to stop sending you these bytes or kick you off their platform altogether. Actually no. If a platform provides a generally available service they are (in many countries, idk. about the US) not allowed to arbitrary exclude some people they don't like without a legal valid reason. And braking legally not valid/binding terms in a ToS is not a legal valid reason. Just because you write something in your ToS doesn't mean it has any legal relevant meaning, there are limits to what you can put in ToS. And limiting (properly done, privacy respecting) research is often not valid. (Through depends a lot on the country.)
  - throwwebmaster 5y ago
    The FTC is a US government entity.
- ClumsyPilot 5y ago
  Imagine I am scraping Twitter - maybe I never accepted their TOS and don't even have an account.
  - throwwebmaster 5y ago
    Interesting that you mentioned Twitter. Twitter requires a user account to access content. Try accessing Twitter while logged out.
throwwebmaster 5y ago
It’s within anyone’s rights as website maintainers to block malicious IP addresses that scrape or otherwise within their discretion. Nobody is legally forcing websites to allow access to everyone, and accordingly, nobody is altering the law by blocking access to people (crawlers, hackers, spammers, malcontents, or anybody really) that they feel are not welcome. So exercising one’s existing rights isn’t an act of making or altering laws. I suggest reading up on what robots.txt is to further understand this.
- ClumsyPilot 5y ago
  Hacking and other malicious behavious are actually illegal. Either Crawling does not belong on that list, or google exects should be in jail. Given that crawling is not malicious, what we are discussing now is 'someone is crawling my website in a way I dont like' which is a different gripe. It mighthave some merit, but robots txt is not legally binding.
  - throwwebmaster 5y ago
    As I stated before, Illegal or not, it is within the website owners’ rights to restrict access.

apazzolini 5y ago

Up until June 14th of this year, the ruling was that scraping is legal from the HiQ vs LinkedIn lawsuit[0]. While finding that link for you, I learned that since that date, the Supreme Court vacated the decision back to the lower courts in light of a new decision of theirs.[1] Now I don't think it's clear one way or the other just yet. Any lawyers here with an opinion on how this is going to go? I haven't found any analysis. [0] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn [1] https://www.reuters.com/technology/us-supreme-court-revives-linkedin-bid-shield-personal-data-2021-06-14/ https://www.reuters.com/technology/us-supreme-court-revives-...

bilbo0s 5y ago
Well given all this and the consent decree, I'm kind of seeing why FB made this decision. If it's legal, let the FTC say so explicitly. It's probably very bad strategy to allow the privacy leak, and then hope that the FTC agrees with your decision later. No one will be sanctioned for adhering too strictly to consent decrees, but you could be sanctioned for being too loose. So the choice is obvious in that light.
- throwwebmaster 5y ago
  It will be interesting to see how the FTC will rectify its ongoing fight to hold FB more accountable for protecting its users’ private data with the FTC’s ostensibly contradictory position to allow mass scraping of private user data at scale in this case. Cambridge Analytica was doing roughly the exact same thing, which was precisely what motivated the FTC's involvement in the first place.
jdminhbg 5y ago
Scraping is legal, but that doesn't mean the site has to make it easy or possible for you. You can scrape my site, but I can also IP ban scrapers.

kcplate 5y ago

“Prohibit” is an interesting word that in the world of public web publishing of data is actually pretty meaningless. Certainly companies take measures to prevent bulk scraping when they can detect it and have legal remedies if copyrighted or owned material is republished. But simply telling me I am prohibited to hold on to a copy of your site’s data is pretty meaningless when my browser caches your page and you want my browser to cache it to speed your site up.

throwwebmaster 5y ago
By accessing the data on a website, you are forming a contract with that website and are subject to the website’s terms and conditions. As a website owner, beyond that, there is no need to tell someone they cannot access your site if you simply block them from accessing your servers instead using a multitude of techniques. Where does caching come into play at all here? You cannot cache content to begin with if the server is blocking access in the first place. And if you have already cached it in the act of violating said website’s terms of service, then you are still not in compliance.
- nieve 5y ago
  Not according to some rulings the last few years. A contract requires a meeting of minds.
  - throwwebmaster 5y ago
    No. According to many rulings the last few years, continually and systematically accessing a third party’s data under the clear expectation that you are aware of (as well as agreed to) their terms is definitely a meeting of minds.
- wkavey 5y ago
  I'm pretty sure that first paragraph is false. Until I explicitly agree to a contract, or "terms and conditions", I am not bound by anything. If a site I navigate to embeds content from another website I am not immediately bound by the terms and conditions of that neclsred site, to think otherwise would invite madness. Not to mention the fact that terms and conditions are not contracts. I don't think they carry the same weight, although someone please correct me on this if I am incorrect.

literallyaduck 5y ago

Are these "researchers" going to be charged the same as the martyr Swartz?

throwwebmaster 5y ago
The parties here would be the US FTC and FB, so no.