6 ms·

> Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a

> Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a second? ToS are subservient to the law; you can (probably) terminate a service account from a user that breaks your ToS, but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS does not apply, since you've technically not entered a binding legal contract with them. > This is almost weirder. If LinkedIn wanted to force users to sign in to view profile info, would they be not allowed to do that because some company had signed a contract that implicitly assumed access to that data? If someone writes a web scraper for my site, and I unknowingly change my site in a way that breaks that scraper, can a court force me to revert the change? IANAL, but I believe that'd fall on intent, and intent is often difficult to prove at a personal level, but not necessarily at a company level. If your intent for putting up barriers that happen to impact scraping, whatever they may be, was indeed to knowingly prevent scraping from a particular company, then you may be liable under this decision. This is the only part of the decision I'm torn on, since it's a bit messy to really prove such things. I'd be much more comfortable with allowing companies to take whatever measures they feel necessary to prevent scraping, and also allowing scrapers to legally circumvent those measures without threat of prosecution, assuming they didn't actually hack into anything.

michaelbuckbee 7y ago
I'm trying to puzzle out how this works in practice. So if LinkedIn has truly public data (no login required to view) then it can be scraped no problem. But if it's only accessible with a login, then it falls under TOS and they can be blocked?
- deleted 7y ago
  [deleted]
andrewmutz 7y ago
> but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS does not apply, since you've technically not entered a binding legal contract with them. Are you sure about this? I am not a lawyer, but I believe that the Terms of Service applies to all users, not just those that explicitly set up a user account. I have interpreted the LinkedIn ruling to mean that scraping public data is no longer criminal activity but it still leaves you open to civil lawsuits for violating the ToS of the website you are scraping.
- rohansingh 7y ago
  > Are you sure about this? I am not a lawyer, but I believe that the Terms of Service applies to all users, not just those that explicitly set up a user account. How would that even work? If I browse to any random public page of your website, it's served to me before you've even transmitted the terms of service. How could I be bound by those terms of service when I haven't even seen them?
  - michaelmior 7y ago
    IANAL, but it seems like ToS could still govern your use of the data which you viewed. Sure, it seems like you couldn't claim any violation based on visiting a random page. But if the ToS is clearly identified on the page and you do something with the data that violates them, perhaps the owner of the site has a case.
    - cabaalis 7y ago
      > perhaps the owner of the site has a case. Except it sounds like the owner doesn't. If the information is on the page made public, the owner of the page can't place terms on what is done with the data downstream. They'd have to implement some real binding system such as authentication where CFAA would apply. (IANAL)
      - turc1656 7y ago
        Correct, but all of that is void if the data presented is any sort of protected information (copyright, IP, etc.). You can't, for example, scrape Yahoo Finance for pricing and dividend history and republish on your own stock tools website. They have a license to redistribute that data and publish on their own website. Similar story for copyrighted text and things of that nature.
        AstralStorm 7y ago
        That would require at least showing that ToS on first use. A link on a page is insufficient. And said ToS would have to force copyright reassignment rather than a general licence, making LinkedIn culpable for any unlawful content published by users of its site.
ChrisMarshallNY 7y ago
There's a long, long history (probably hundreds –if not thousands– of years old) of selling aggregated or processed publicly-available information. I'm not particularly thrilled with it, but enough people think of it as a valuable enough service to pay for; even if they know they could get it themselves, for free. LinkedIn users (as opposed to the company) might actually like what HiQ is doing, as it may help their own prospects.
- ivanhoe 7y ago
  > but enough people think of it as a valuable enough service to pay for; even if they know they could get it themselves, for free. It's not free, it takes time to collect data. Buying it makes a lot of sense as long as you pay less than what's your own time worth to you...
  - Cerium 7y ago
    It is true in the current situation, though I would prefer that we ensure free data must be free. In that case buyers of data would be incentivized to pressure providers of free data to improve the data quality.
    - moron4hire 7y ago
      The data does remain free, as long as LinkedIn still provides it for free. The data without the noise is what you're paying for. The service of winnowing out what you care about from what you don't care about. Considering how big of an effort it is, and that the source from which it came is still available, why should the cleaned data be free? If I collect fallen trees from public land, chop it into usable firewood, should my bundles of firewood also be free? Or I collect solar power with my own solar cells, should I have to give you the electricity for free?
      - ChrisMarshallNY 7y ago
        Absolutely spot-on. I'm thinking of processed GIS data. If you have ever tried using the various formats that are supplied by government sites, you know what a huge pain it is. I'm happy to pay a reasonable price for an interpreted and bowdlerized version.

michaelbuckbee 7y ago

I'm trying to puzzle out how this works in practice. So if LinkedIn has truly public data (no login required to view) then it can be scraped no problem. But if it's only accessible with a login, then it falls under TOS and they can be blocked?

deleted 7y ago
[deleted]

andrewmutz 7y ago

> but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS does not apply, since you've technically not entered a binding legal contract with them. Are you sure about this? I am not a lawyer, but I believe that the Terms of Service applies to all users, not just those that explicitly set up a user account. I have interpreted the LinkedIn ruling to mean that scraping public data is no longer criminal activity but it still leaves you open to civil lawsuits for violating the ToS of the website you are scraping.

rohansingh 7y ago
> Are you sure about this? I am not a lawyer, but I believe that the Terms of Service applies to all users, not just those that explicitly set up a user account. How would that even work? If I browse to any random public page of your website, it's served to me before you've even transmitted the terms of service. How could I be bound by those terms of service when I haven't even seen them?
- michaelmior 7y ago
  IANAL, but it seems like ToS could still govern your use of the data which you viewed. Sure, it seems like you couldn't claim any violation based on visiting a random page. But if the ToS is clearly identified on the page and you do something with the data that violates them, perhaps the owner of the site has a case.
  - cabaalis 7y ago
    > perhaps the owner of the site has a case. Except it sounds like the owner doesn't. If the information is on the page made public, the owner of the page can't place terms on what is done with the data downstream. They'd have to implement some real binding system such as authentication where CFAA would apply. (IANAL)
    - turc1656 7y ago
      Correct, but all of that is void if the data presented is any sort of protected information (copyright, IP, etc.). You can't, for example, scrape Yahoo Finance for pricing and dividend history and republish on your own stock tools website. They have a license to redistribute that data and publish on their own website. Similar story for copyrighted text and things of that nature.
      - AstralStorm 7y ago
        That would require at least showing that ToS on first use. A link on a page is insufficient. And said ToS would have to force copyright reassignment rather than a general licence, making LinkedIn culpable for any unlawful content published by users of its site.

ChrisMarshallNY 7y ago

There's a long, long history (probably hundreds –if not thousands– of years old) of selling aggregated or processed publicly-available information. I'm not particularly thrilled with it, but enough people think of it as a valuable enough service to pay for; even if they know they could get it themselves, for free. LinkedIn users (as opposed to the company) might actually like what HiQ is doing, as it may help their own prospects.

ivanhoe 7y ago
> but enough people think of it as a valuable enough service to pay for; even if they know they could get it themselves, for free. It's not free, it takes time to collect data. Buying it makes a lot of sense as long as you pay less than what's your own time worth to you...
- Cerium 7y ago
  It is true in the current situation, though I would prefer that we ensure free data must be free. In that case buyers of data would be incentivized to pressure providers of free data to improve the data quality.
  - moron4hire 7y ago
    The data does remain free, as long as LinkedIn still provides it for free. The data without the noise is what you're paying for. The service of winnowing out what you care about from what you don't care about. Considering how big of an effort it is, and that the source from which it came is still available, why should the cleaned data be free? If I collect fallen trees from public land, chop it into usable firewood, should my bundles of firewood also be free? Or I collect solar power with my own solar cells, should I have to give you the electricity for free?
    - ChrisMarshallNY 7y ago
      Absolutely spot-on. I'm thinking of processed GIS data. If you have ever tried using the various formats that are supplied by government sites, you know what a huge pain it is. I'm happy to pay a reasonable price for an interpreted and bowdlerized version.