23 ms·

9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

1136 points by donohoe 7y ago

mullingitover 7y ago
> LinkedIn has taken steps to protect the data on its website from what it perceives as misuse or misappropriation. The instructions in LinkedIn’s “robots.txt” file—a text file used by website owners to communicate with search engine crawlers and other web robots—prohibit access to LinkedIn servers via automated bots, except that certain entities, like the Google search engine, have express permission from LinkedIn for bot access. Not a big fan of weev, but this sure seems like he got screwed if he was just enumerating public web pages and went to jail for it.[1] [1] https://en.wikipedia.org/wiki/Weev#AT&T_data_breach https://en.wikipedia.org/wiki/Weev#AT&T_data_breach
- tick_tock_tick 7y ago
  Not saying the court made the right call but for that case the big issue for the court was the pages were clearly not intended for the public and the defendant knew it.
  - Kalium 7y ago
    I believe the salient issue is whether or not there were effective access controls, not whether or not a page could be reasonably interpreted as intended to be non-public.
    - henryfjordan 7y ago
      The real issue here is somewhere between both you and GP. What is required to trigger the CFAA? Does accessing a page the site owner doesn't want you to violate the CFAA or do you need to hack through access controls?
      - akersten 7y ago
        And does robots.txt count as an access control? What about a humans.txt that says "please don't scrape this site"?
        blackflame7000 7y ago
        Sounds more like an access suggestion than control to me
        tempestn 7y ago
        According to the ruling, a cease and desist letter directly demanding that they not scrape the site didn't count as access control, so one would assume that humans.txt wouldn't either. It needs to be a technical prevention like a password, access token, etc.
      - txcwpalpha 7y ago
        As a real-world analogue: you can indeed be guilty of trespassing on someone's property even if you don't have to jump over any fences or pick any locks to get there. In some places, they don't even have to have a "no trespassing" sign. Simply being present on someone else's property without an invitation from them is illegal, and no, an open door does not count as an invitation.
        LyndsySimon 7y ago
        You generally can’t be charged with trespass unless you refuse to leave when told to do so. An open door to a home is different, but unfenced property is 100% not trespass until you refuse to leave.
        txcwpalpha 7y ago
        That's not true at all. If you are aware that the property you are accessing is not meant for your use, you can be charged with trespassing regardless of if you have specifically been asked to leave or not. It's even possible to be guilty of trespass even if you weren't aware that you weren't allowed on the land. This is negligent trespassing.
    - rayiner 7y ago
      There is no requirement to have "effective access controls." What matters is what a reasonable person would believe about whether they were allowed to access the data. The access controls are relevant only insofar as that they convey a message that access is not permitted. The effectiveness of the access control is utterly irrelevant.
    - momokoko 7y ago
      That is false. What effective access controls do, legally speaking, is help determine if a person could reasonably conclude that the information was non-public. For example, if a door is locked, but easily defeated, there is an implied assumption that what lies behind it is only available to someone with the key. Another example is an unlocked door with a sign that states "No access without authorization". Or an unmarked and unlocked door that is on private property in a place where it would be very unlikely for someone to reasonably conclude that the access was intended to be public.
  - blackflame7000 7y ago
    Ahh so if a company leaks data it's the viewer's fault, not the companies?
    - rayiner 7y ago
      Yes. That's the general rule--negligence of a victim does not negate the culpability of the criminal. "It was easy to commit the crime" is not a defense. If you find yourself with access to something you think you're not supposed to have access to, you're supposed to do the right thing.
      - deleted 7y ago
        [deleted]
      - manfredo 7y ago
        I think the issue is not "negligence of the victim" so much as, "took no steps to make the information private". If I don't lock my door and someone goes into my house, opens my filing cabinet, and copies my financial info then they're still a criminal. They took information from a place that was unambiguously meant to be private. But if I staple a copy of my financial info to a telephone pole, surely those who read it are not committing the crime of reading private information. A URL that is not authenticated seems more like the latter than the former. The web is public unless people take steps to make it private. Criminalizing accessing unprotected URLs is like arresting people for reading the financial info I left stapled to the telephone pole. A while back there was a "world's most exclusive chat room" website that hosted IRC channels for people above twitter follower numbers. People learned that they could just manually increment a URL parameter to access higher room numbers regardless of their twitter follower count. Were those people committing a crime?
        tptacek 7y ago
        To take the case at issue, when your immediate reaction to discovering an unprotected URL is to scrape it, discuss on an IRC channel how you're going to monetize it, and then go to the media to announce your security vulnerability discovery, you are going to find it difficult to make the argument that you believed you had authorized access.
- UncleMeat 7y ago
  A friend of mine from grad school was very involved in legal issues related to cfaa stuff. According to him, weev really got screwed because he failed "the punk test", which discouraged lawyers from wanting to use him as a test case.
  - hitekker 7y ago
    Curious question: what is "the punk test"?
    - jakear 7y ago
      “Would this person’s attitude make them more unpleasant to work with than others I could be representing in my already limited time”, or something to that effect. Someone who spends their free time hacking university printers to distribute white supremacist propaganda and is a proud member of the “Gay Nigger Association of America” trolling group would likely not pass.
    - throwawayjava 7y ago
      "will this person be negatively perceived by a jury of their peers, or, especially, a jury of 3 federal judge? And, will this person say dumb shit that torpedos my case for reasons other than its merits"
    - UncleMeat 7y ago
      "Is this person going to say idiotic nonsense that tanks the case because the judge or jury thinks they are just awful?" Weev may have been a good test case if he wasn't a white supremacist.
- lwf 7y ago
  Even if the case was tried today, 9th Cir. isn't binding on other regions of the US, and there's a bit of a split, as detailed in the opinion[1]: > In recognizing that the CFAA is best understood as an anti-intrusion statute and not as a “misappropriation statute,” we rejected the contract-based interpretation of the CFAA’s “without authorization” provision adopted by some of our sister circuits. Compare Facebook, Inc. v. Power Ventures, Inc., 844 F.3d 1058, 1067 (9th Cir. 2016), cert. denied, 138 S. Ct. 313 (2017) (“[A] violation of the terms of use of a website—without more— cannot establish liability under the CFAA.”); Nosal I, 676 F.3d at 862 (“We remain unpersuaded by the decisions of our sister circuits that interpret the CFAA broadly to cover violations of corporate computer use restrictions or violations of a duty of loyalty.”), with EF Cultural Travel BV v. Explorica, Inc., 274 F.3d 577, 583–84 (1st Cir. 2001) (holding that violations of a confidentiality agreement or other contractual restraints could give rise to a claim for unauthorized access under the CFAA); United States v. Rodriguez, 628 F.3d 1258, 1263 (11th Cir. 2010) (holding that a defendant “exceeds authorized access” when violating policies governing authorized use of databases). weev was tried in an area under the 3rd Cir. jurisdiction. Somewhat interestingly, his conviction was thrown out in 2014 on venue grounds (e.g. being tried in NJ), without addressing the statutory question.[2] [1]: pp. 27-28 [2]: https://en.wikipedia.org/wiki/Weev?oldid=912921723#cite_ref-48 https://en.wikipedia.org/wiki/Weev?oldid=912921723#cite_ref-...
  - joncrane 7y ago
    Is there an AWS region in the District governed by this case, so you can just do all your web scraping from instances in that region?
- devmunchies 7y ago
  >except that certain entities, like the Google search engine, have express permission from LinkedIn for bot access how does this work technically? i just tried crawling a friend's profile using curl and set my user agent to Google's bot and it still was blocked.
  - phs318u 7y ago
    IP range whitelisting? MASSL? EDITED: corrected auto-correct.
  - dx034 7y ago
    Google describes how to verify Googlebot here: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en Most other search engine crawlers provide similar methods.
hartator 7y ago
That’s awesome news. Thanks also to the EFF for all the work they are doing to ensure fair use is still a thing. We’ll (https://serpapi.com https://serpapi.com) be donating next year.
henryfjordan 7y ago
hiQ asked the court for a preliminary injunction to stop Linkedin from denying them access, won it, and this is the result of Linkedin's appeal of that injunction. This is not the end of the case. The title is wrong. The 9th Circuit just ruled that hiQ has a decent enough argument to move forward. The question of whether them scraping a public site can violate the CFAA is not settled. > We therefore conclude that hiQ has raised a serious question as to whether the reference to access “without authorization” limits the scope of the statutory coverage to computer information for which authorization or access permission, such as password authentication, is generally required > The data hiQ seeks to access is not owned by LinkedIn and has not been demarcated by LinkedIn as private using such an authorization system. HiQ has therefore raised serious questions about whether LinkedIn may invoke the CFAA to preempt hiQ’s possibly meritorious tortious interference claim. Note the tone of the language used in the ruling. The judge makes it pretty clear that nothing is final here.
- foota 7y ago
  I disagree about the tone, it seems to suggest to me that the judge believes there is a strong case here for hiQ.
  - henryfjordan 7y ago
    I agree, but there's a difference between "hiQ has a strong case" and "Here is the final ruling on the hiQ case". I was trying to point out that the case is not over or ruled on at all. Just the preliminary injunction.
    - deleted 7y ago
      [deleted]
    - AznHisoka 7y ago
      "Just the preliminary injunction." But I thought that was a few years ago! How many more over-rulings, or appeals do we freaking need? I really hope this is the final ruling.
payne92 7y ago
Flagged for misleading title.
- akersten 7y ago
  Would you mind suggesting a more accurate title?
  - btown 7y ago
    9th Circuit issues injunction preventing LinkedIn from blocking hiQ's scraping of public pages (IANAL but this seems like a reasonable summary of the first paragraph of the document.)
akersten 7y ago
A good decision was reached, but it's a little worrying that the emphasis in the ruling was mostly about a weighing of business interests rather than affirming a right to access public information. If HiQ's business model had not been jeopardized by LinkedIn's business desire to block them, I fear this court could have easily gone the other way. I'd really love to see a ruling that solidifies the right of someone to access publicly available data without fear of repercussions. If this case makes it to SCOTUS, I would hope the ruling is predicated on that rather than business harm. Key paragraphs from the ruling: > In short, even if some users retain some privacy interests in their information notwithstanding their decision to make their profiles public, we cannot, on the record before us, conclude that those interests—or more specifically, LinkedIn’s interest in preventing hiQ from scraping those profiles—are significant enough to outweigh hiQ’s interest in continuing its business, which depends on accessing, analyzing, and communicating information derived from public LinkedIn profiles. > Nor do the other harms asserted by LinkedIn tip the balance of harms with regard to preliminary relief. LinkedIn invokes an interest in preventing “free riders” from using profiles posted on its platform. But LinkedIn has no protected property interest in the data contributed by its users, as the users retain ownership over their profiles. And as to the publicly available profiles, the users quite evidently intend them to be accessed by others, including for commercial purposes—for example, by employers seeking to hire individuals with certain credentials. Of course, LinkedIn could satisfy its “free rider” concern by eliminating the public access option, albeit at a cost to the preferences of many users and, possibly, to its own bottom line.
- victor9000 7y ago
  > If HiQ's business model had not been jeopardized I think this is more about validating hiQ's legal standing in the case.
Animats 7y ago
This action does more than that. The court left the preliminary injunction against LinkedIn in place: "The district court granted hiQ’s motion. It ordered LinkedIn to withdraw its cease-and-desist letter, to remove any existing technical barriers to hiQ’s access to public profiles, and to refrain from putting in place any legal or technical measures with the effect of blocking hiQ’s access to public profiles." So LinkedIn is prohibited from blocking hiQ's access by technical means. That's a strong holding. If this case is eventually decided in favor of hiQ, scrapers can no longer be blocked. Throttled a little, maybe, but no more than other users doing a comparable query rate.
- buro9 7y ago
  .
  - blacksmith_tb 7y ago
    Hmm, not a lawyer, but I think if you commit one crime in the process of doing something else that's legal, you're still guilty. But LinkedIn couldn't block your spider if it came back and throttled at a reasonable rate.
  - thomasjudge 7y ago
    In the opinion they specifically discuss DOS attacks: "Internet companies and the public do have a substantial interest in thwarting denial-of-service attacks and blocking abusive users, identity thieves, and other ill-intentioned actors. But we do not view the district court’s injunction as opening the door to such malicious activity"
- emilfihlman 7y ago
  Leaving the injunction in place is insane and a huge oversight. It amounts to making web pages carriers that cannot select who they serve. It should have said only that there is nothing judicially wrong with scraping but also not limited the rights of a service.
  - dhimes 7y ago
btown 7y ago
A choice quote: > In recognizing that the CFAA is best understood as an anti-intrusion statute and not as a “misappropriation statute,” Nosal I, 676 F.3d at 857–58, we rejected the contract-based interpretation of the CFAA’s “without authorization” provision adopted by some of our sister circuits. Compare Facebook, Inc. v. Power Ventures, Inc., 844 F.3d 1058, 1067 (9th Cir. 2016), cert. denied, 138 S. Ct. 313 (2017) (“[A] violation of the terms of use of a website—without more— cannot establish liability under the CFAA.”); Nosal I, 676 F.3d at 862 (“We remain unpersuaded by the decisions of our sister circuits that interpret the CFAA broadly to cover violations of corporate computer use restrictions or violations of a duty of loyalty.”), with EF Cultural Travel BV v. Explorica, Inc., 274 F.3d 577, 583–84 (1st Cir. 2001) (holding that violations of a confidentiality agreement or other contractual restraints could give rise to a claim for unauthorized access under the CFAA); United States v. Rodriguez, 628 F.3d 1258, 1263 (11th Cir. 2010) (holding that a defendant “exceeds authorized access” when violating policies governing authorized use of databases). And: > As one prominent commentator has put it, “an authentication requirement, such as a password gate, is needed to create the necessary barrier that divides open spaces from closed spaces on the Web.” Orin S. Kerr, Norms of Computer Trespass, 116 Colum. L. Rev. 1143, 1161 (2016). Moreover, elsewhere in the statute, password fraud is cited as a means by which a computer may be accessed without authorization, see 18 U.S.C. § 1030(a)(6),12 bolstering the idea that authorization is only required for password-protected sites or sites that otherwise prevent the general public from viewing the information. My layman's (IANAL) interpretation of preliminary injunctions are that the case is far from over, and this could be overturned at any time as more deliberation is done (assuming LinkedIn wants to keep throwing money at that relatively slim possibility). But now this research has been done, and future courts have the ability to look to these references and lines of reasoning all in one place. EDIT: https://twitter.com/OrinKerr/status/1171116153948626944?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet https://twitter.com/OrinKerr/status/1171116153948626944?ref_... is an analysis by the law professor cited above: > BIG NEWS: 9th Circuit holds that scraping a public website likely does not violate the CFAA, even after website owner prohibits with a cease-and-desist letter; language strongly suggests CFAA only applies to bypassing authentication. Blog post up soon. http://cdn.ca9.uscourts.gov/datastore/opinions/2019/09/09/17-16783.pdf http://cdn.ca9.uscourts.gov/datastore/opinions/2019/09/09/17... #N
- 7y ago
dx87 7y ago
Would that ruling mean that sites could no longer refuse to show content based on how they're accessed? For example, sites that won't load if the browser is in headless mode, or sites that depend on javascript as a way of blocking wget/curl.
- freehunter 7y ago
  I have a scraper for a site that used to offer an API for their publicly available site but removed the API with no warning. The info is still available to the general public, but only through their website. I created a scraper for the public page, but shortly after they switched to loading some public information through Javascript so my HTML scraper couldn't see it anymore. I ended up having to write an application around Selenium to load the Javascript and import this public information. I'm just waiting for them to start randomizing the CSS classes to make scraping even harder. The content is static, even as data changes on the server it does not refresh on the page unless you reload the page. There is no reason why your page should refuse to load plain text without Javascript enabled.
  - Pfhreak 7y ago
    > There is no reason why your page should refuse to load plain text without Javascript enabled. Sure there is. You prefer writing javascript and you want to serve your site through a CDN. You might not think that's a good reason, but that's certainly a reason.
    - Franciscouzo 7y ago
      What does a CDN has to do with it?
      - Spivak 7y ago
        Because if your application bundle is a fixed asset — like a JS SPA that fetches it’s data from an API then you can distribute your entire application via an inexpensive CDN. As soon as your application bundle is rendered on your servers dynamically then only part of your site can be delivered via CDN. Basically going all-JS gives you an app model where your sever side code doesn’t even know or care about HTML or the web or whatever. It just pushes JSON or whatever around and it largely client independent. Great model when you need to support iOS, Android, web, desktop.
kevin_b_er 7y ago
Best hope that hiQ prevails. The slope slips very fast without LinkedIn's defeat. If LinkedIn prevails, the "EULA" has the force of criminal law and not just an agreement that lacks the meeting of the minds.
meowface 7y ago
Considering the kind of private scraping and selling tactics LinkedIn has been chronically guilty of (and not just the ordinary "growth hack" stuff: "LinkedIn violated data protection by using 18M email addresses of non-members to buy targeted ads on Facebook" [1]), it's satisfying to see LinkedIn lose this. [1] https://techcrunch.com/2018/11/24/linkedin-ireland-data-protection/ https://techcrunch.com/2018/11/24/linkedin-ireland-data-prot...
- penagwin 7y ago
  I feel like this is a really common theme I've seen several times. Something like "Music Lyric site X sues Google for embedding their lyrics in the results directly" which is funny because site X got the lyrics by scraping them from other sites. Plus Google only exists from scraping content, but I believe their TOS includes "don't scrape our content". I find it really funny that the scrapers are battling scrapers - like guys you only exist because you do THE EXACT SAME THING
  - dmix 7y ago
    Regardless, there is legitimate value in the collection, cleaning, interlinking, and presentation of existing data. How that is interpreted by the law is one thing but merely because the data came from a variety of other public/private sources doesn't mean it derived all of its value externally.
    - meowface 7y ago
      For sure, but they shouldn't be hypocritical about it. If they don't consider themselves content parasites, they shouldn't consider people scraping their site to be content parasites, either. (Some sites really are just parasites, though.)
      - dodobirdlord 7y ago
        There's nothing hypocritical about it. Googlebot respects robots.txt configured on pages it scrapes. Google in turn expects that their own robots.txt will be respected. What's the issue? https://www.google.com/robots.txt https://www.google.com/robots.txt
meowface 7y ago
Considering the kind of private scraping and selling tactics LinkedIn has been chronically guilty of (and not just the ordinary "growth hack" stuff: "LinkedIn violated data protection by using 18M email addresses of non-members to buy targeted ads on Facebook" [1]), it's satisfying to see LinkedIn lose this. [1] https://techcrunch.com/2018/11/24/linkedin-ireland-data-protection/ https://techcrunch.com/2018/11/24/linkedin-ireland-data-prot...
codedokode 7y ago
This is actually bad, would not it be better if sites would be allowed to block crawlers? I don't see what is the legal basis for forbidding to ban scrapers. Is there a law that a site must serve pages for anyone?
- CamperBob2 7y ago
  Parent was downvoted but I think they have a point. This sounds overbroad, to an extent that I'd worry will get the whole ruling tossed out by SCOTUS. On the other hand, if the ruling stands, it sounds like it will finally be possible to do useful things with Craigslist.
- anortef 7y ago
  Why? if your data is publicly accessible what is the difference between using scripts and someone hiring a ton of people on a third world country to copy&paste your content?
- cooljacob204 7y ago
  Because they are exposing their website to the general public. If you don't want the public to have access and be able to scrap it don't make it public facing.
- 3xblah 7y ago
  There is no legal basis for "forbidding to ban scrapers". The question is whether there is any legal basis for banning scrapers, i.e., for blocking hiQ. In other words, if hiQ keeps scraping, are they violating anyone's rights and/or breaking the law by doing that? As long as that remains a legitimate, open question, then hiQ can argue they should be allowed to keep scraping without incurring civil or criminal liability. That is the purpose of the injunction. There could be no legal basis for blocking hiQ. Until that question is resolved, hiQ can keep on scraping.
- xxxpupugo 7y ago
  You can redefine what is 'public', or require login to see those information anyway. If anything, the ruling could just push websites to hide information even deeper.
CosmicShadow 7y ago
Even if LinkedIn loses and scrapers can no longer be blocked, they still just switched to putting all profiles behind an authwall, or at least it's very hard to not get an authwall. So could HiQ even carry on if they won anyway?
- jolmg 7y ago
  I'm not very familiar with neither LinkedIn nor HiQ, but what would be the problem with logging in before scraping?
  - henryfjordan 7y ago
    The reason the pages are public to begin with is that Google will only scrape public pages for search indexing. LinkedIn wants to provide the pages ONLY to google, so they tried telling hiQ to stop scraping without any physical blockers (so as to not impede google's scraper). If LinkedIn loses this case they (and others) might try to get Google to change their policy (either use auth or some whitelisted IP addresses or something).
inbland 7y ago
What if hiQ were to then go and sell the information that's been scraped from public profiles? Would this be considered illegal?
- colechristensen 7y ago
  I would assume the data would still be covered by copyright meaning they could use that data and maybe create and sell derivative works, but not just scrape and publish.
  - icebraining 7y ago
    Pictures and such are covered by copyright, but mere facts are not (at least in the US): https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._Rural_Telephone_Service_Co https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
    - baroffoos 7y ago
      Collections of facts are copyrightable. Which is why you can go out and make a map of your local area but you may not copy the data from google maps. You may end up with the exact same data and that is ok because you both copied the same facts but if there is a mistake on google maps (Perhaps placed as a trap) then you can be caught if your map has the same mistake.
      - icebraining 7y ago
        Only the creative aspects of those collections of facts are copyrightable. You can definitively copy from Google Maps, just like Feist could copy from Rural; what you can't copy are the aspects of the maps that require creativity. As for trap streets, they don't help as much as you think; as we've seen, simply showing that copying occurred is not enough. There was a theory that the trap streets - since they were invented, not facts - could themselves be copyrighted, but in Alexandria Drafting Co. v. Amsterdam, the courts said that copying a few trap streets among a bunch of facts was too minimal to be considered infringement.
mirimir 7y ago
This is a great opinion. But the 9th circuit is arguably still ~atypical.
supernova87a 7y ago
This case is so ridiculous on multiple fronts that although this procedural ruling (injunction) seems technically correct (to allow the case to proceed to actual court), it could just as well have been thrown out with no difference in or ultimate harm to the parties. First, LinkedIn makes the claim that its users have a right to privacy against scraping by such a 3rd party. That's laughable. As the court saw, their whole business model is made on people sharing their profiles broadly and mostly to the public. Secondly, HiQ claims that LinkedIn's efforts to stop it from using the data are tortious interference. That's bold -- suppose someone is taking your assets (you believe illegally) and selling them to others -- can you imagine the gall that the person taking your assets can sue you for interfering with their subsequent sale of your assets? Finally, that LinkedIn resorted to using the computer fraud and anti-terrorism statutes to make their argument is ridiculous. So much craziness to go around. I would've just tossed the case, but I guess there is the whole bit about due process... Maybe HiQ will fail anyway at the next substantive trial, but what a waste of time.
- judge2020 7y ago
  > suppose someone is taking your assets Except that, in the digital sense, it's only copied. They now have it, but you didn't lose your assets or money besides the <$0.001 it costs to serve each web page. > So much craziness to go around. I agree - I haven't read through the entire thing, but it looks like, instead of saying "you can't scrape", they could implicity give a license to users for personal and business use, but not be allowed the reselling of the data (of course carefully worded to allow the likes of Recruiters and whatnot to do so). It's like trying to argue that the DMCA says you can't create a torrent file of some movie.
victor9000 7y ago
On how this case relates to the CFAA: We therefore conclude that hiQ has raised a serious question as to whether the reference to access “without authorization” limits the scope of the statutory coverage to computer information for which authorization or access permission, such as password authentication, is generally required. Put differently, the CFAA contemplates the existence of three kinds of computer information: (1) information for which access is open to the general public and permission is not required, (2) information for which authorization is required and has been given, and (3) information for which authorization is required but has not been given (or, in the case of the prohibition on exceeding authorized access, has not been given for the part of the system accessed). Public LinkedIn profiles, available to anyone with an Internet connection, fall into the first category. With regard to such information, the “breaking and entering” analogue invoked so frequently during congressional consideration has no application, and the concept of “without authorization” is inapt.
cwkoss 7y ago
In tangentially related news, Reid Hoffman, LinkedIn co-founder, spent some time last week defending Ito's connections to pedophile Epstein. https://twitter.com/anandwrites/status/1169952673988300800 https://twitter.com/anandwrites/status/1169952673988300800
sebastianconcpt 7y ago
In short, even if some users retain some privacy interests in their information notwithstanding their decision to make their profiles public, we cannot, on the record before us, conclude that those interests—or more specifically, LinkedIn’s interest in preventing hiQ from scraping those profiles—are significant enough to outweigh hiQ’s interest in continuing its business, which depends on accessing, analyzing, and communicating information derived from public LinkedIn profiles. Reasonable. If a platform helps you make information of an individual public, then why it should matter for the platform how the market uses that public information?
sheeshkebab 7y ago
Great, finally. Don’t pretend the information is public and then block access to it. It’s either private and inaccessible, or it’s free for all to use.
deleted 7y ago
[deleted]
ummonk 7y ago
Does this have implications for other scraping? (E.g. https://techcrunch.com/2019/05/20/instagram-influencer-celebrity-accounts-scraped/ https://techcrunch.com/2019/05/20/instagram-influencer-celeb...)
- sometinsome 7y ago
  Or this other case https://gizmodo.com/instagram-boots-ad-partner-hyp3r-for-reportedly-scrapin-1837051804 https://gizmodo.com/instagram-boots-ad-partner-hyp3r-for-rep...
OrgNet 7y ago
I got 80,000 recipes from allrecipes.com... glad to know it was legal
asah 7y ago
Congrats Darren! Can't wait for your next one to come out of stealth mode.
dzonga 7y ago
how about websites using nonsense css-classes usually autogenerated through frameworks that make scrapping difficult ? I'm sure this ruling doesn't cover that case ? well globally I wish authorities would rule that public data should be published in computer accessible format e.g pdf's for humans and xblr's / csv for machines e.g in financial reports. lots of data in pdf's that costs a ton to mine. & tools like AWS Textract are hardly up to task.
cookie_monsta 7y ago
So the champion of the public internet turns out to be a company that scrapes your social media, MLs it and sells the results to your HR dept? I'm reminded of Dave Chapelle's Halle Berry routine...
anthony_doan 7y ago
Sweet, I scraped a bit of data and have been holding off on putting it on to my web app. I think I'll transform the data a bit before putting it up there.
monksy 7y ago
RIP Aaron Swartz
nirse 7y ago
I have never heard of a 'cause-and-desist' letter before. Odd spelling mistake for such a document.
thuylun2412 7y ago
nice
Shivetya 7y ago
Volokh take is an interesting read [1] I am curious how quickly most pages get put behind authorization. With the wording of this ruling you could pretty much go snap up any blog side (say like medium) and more. I wonder what kind of services would come out of that, having the data in a format it can be more easier parsed/analyzed? so every ecommerce site is fair game? I assume most are already being scraped but I cannot imagine having to be in an environment where many of your connections are not people [1] https://reason.com/2019/09/09/scraping-a-public-website-doesnt-violate-the-cfaa-ninth-circuit-mostly-holds/ https://reason.com/2019/09/09/scraping-a-public-website-does...
- ankurkwv 7y ago
  Hmm - I think a key in the ruling here was that LinkedIn maintains no copyright claim on these pages. Users on LinkedIn retain ownership of their profile data. Compare that to a blog and maybe copyright could come into play? Not a lawyer just thinking out loud...
sha666sum 7y ago
This is a weird case, as it turns the question of scraping on its head. Normally you'd think "am I allowed to scrape?", but instead the question becomes "am I allowed to prevent scraping?". Anyways, I disagree with the court's judgment here. The users have consented that their data be used in accordance with LinkedIn's privacy policy. Even if it is publicly posted does not mean that the user has relinquished control over their personal information for another company to do with as they wish.
prirun 7y ago
eBay had better lawyers than LinkedIn: https://casetext.com/case/ebay-v-bidders-edge https://casetext.com/case/ebay-v-bidders-edge I'm glad this court ruled it wasn't a violation of CFAA. But using trespass to prevent it seems reasonable. A private business should be allowed to restrict certain kinds of use of its resources (servers, bandwidth, etc), especially if it is beyond typical use. But if the load is typical and doesn't actually harm LinkedIn, it seems less reasonable to restrict them. If LinkedIn doesn't want automated access to their data because it is too much of a load on their servers, then they should be required to ban ALL automated access, including Google's bots. Of course they want Google's bots because that sends them traffic. Another reason I think it was stupid for LinkedIn to use CFAA is that it sets them up to be a protected computer system, with protected information. If that is the case, it seems they could be liable for disclosing the information to someone a user didn't want, like a stalked. It's rather dumb: LI is claiming they host protected information, but it is only protected against someone that might compete with them.
- silentguy 7y ago
  > If LinkedIn doesn't want automated access to their data because it is too much of a load on their servers, then they should be required to ban ALL automated access, including Google's bots. Of course they want Google's bots because that sends them traffic. By that logic, I should have access to linked premium features for free. Why should linkedin give more data to people who pay?
V3ritas1337 7y ago
Of course it isn't, it's known as implicit authorisation.
fulldecent2 7y ago
LinkedIn is a public website? Every time I click from DDG it shows a login page and zero content.

1136 points by donohoe 7y ago

> LinkedIn has taken steps to protect the data on its website from what it perceives as misuse or misappropriation. The instructions in LinkedIn’s “robots.txt” file—a text file used by website owners to communicate with search engine crawlers and other web robots—prohibit access to LinkedIn servers via automated bots, except that certain entities, like the Google search engine, have express permission from LinkedIn for bot access. Not a big fan of weev, but this sure seems like he got screwed if he was just enumerating public web pages and went to jail for it.[1] [1] https://en.wikipedia.org/wiki/Weev#AT&T_data_breach https://en.wikipedia.org/wiki/Weev#AT&T_data_breach

tick_tock_tick 7y ago
Not saying the court made the right call but for that case the big issue for the court was the pages were clearly not intended for the public and the defendant knew it.
- Kalium 7y ago
  I believe the salient issue is whether or not there were effective access controls, not whether or not a page could be reasonably interpreted as intended to be non-public.
  - henryfjordan 7y ago
    The real issue here is somewhere between both you and GP. What is required to trigger the CFAA? Does accessing a page the site owner doesn't want you to violate the CFAA or do you need to hack through access controls?
    - akersten 7y ago
      And does robots.txt count as an access control? What about a humans.txt that says "please don't scrape this site"?
      - blackflame7000 7y ago
        Sounds more like an access suggestion than control to me
      - tempestn 7y ago
        According to the ruling, a cease and desist letter directly demanding that they not scrape the site didn't count as access control, so one would assume that humans.txt wouldn't either. It needs to be a technical prevention like a password, access token, etc.
    - txcwpalpha 7y ago
      As a real-world analogue: you can indeed be guilty of trespassing on someone's property even if you don't have to jump over any fences or pick any locks to get there. In some places, they don't even have to have a "no trespassing" sign. Simply being present on someone else's property without an invitation from them is illegal, and no, an open door does not count as an invitation.
      - LyndsySimon 7y ago
        You generally can’t be charged with trespass unless you refuse to leave when told to do so. An open door to a home is different, but unfenced property is 100% not trespass until you refuse to leave.
        txcwpalpha 7y ago
        That's not true at all. If you are aware that the property you are accessing is not meant for your use, you can be charged with trespassing regardless of if you have specifically been asked to leave or not. It's even possible to be guilty of trespass even if you weren't aware that you weren't allowed on the land. This is negligent trespassing.
  - rayiner 7y ago
    There is no requirement to have "effective access controls." What matters is what a reasonable person would believe about whether they were allowed to access the data. The access controls are relevant only insofar as that they convey a message that access is not permitted. The effectiveness of the access control is utterly irrelevant.
  - momokoko 7y ago
    That is false. What effective access controls do, legally speaking, is help determine if a person could reasonably conclude that the information was non-public. For example, if a door is locked, but easily defeated, there is an implied assumption that what lies behind it is only available to someone with the key. Another example is an unlocked door with a sign that states "No access without authorization". Or an unmarked and unlocked door that is on private property in a place where it would be very unlikely for someone to reasonably conclude that the access was intended to be public.
- blackflame7000 7y ago
  Ahh so if a company leaks data it's the viewer's fault, not the companies?
  - rayiner 7y ago
    Yes. That's the general rule--negligence of a victim does not negate the culpability of the criminal. "It was easy to commit the crime" is not a defense. If you find yourself with access to something you think you're not supposed to have access to, you're supposed to do the right thing.
    - deleted 7y ago
      [deleted]
    - manfredo 7y ago
      I think the issue is not "negligence of the victim" so much as, "took no steps to make the information private". If I don't lock my door and someone goes into my house, opens my filing cabinet, and copies my financial info then they're still a criminal. They took information from a place that was unambiguously meant to be private. But if I staple a copy of my financial info to a telephone pole, surely those who read it are not committing the crime of reading private information. A URL that is not authenticated seems more like the latter than the former. The web is public unless people take steps to make it private. Criminalizing accessing unprotected URLs is like arresting people for reading the financial info I left stapled to the telephone pole. A while back there was a "world's most exclusive chat room" website that hosted IRC channels for people above twitter follower numbers. People learned that they could just manually increment a URL parameter to access higher room numbers regardless of their twitter follower count. Were those people committing a crime?
      - tptacek 7y ago
        To take the case at issue, when your immediate reaction to discovering an unprotected URL is to scrape it, discuss on an IRC channel how you're going to monetize it, and then go to the media to announce your security vulnerability discovery, you are going to find it difficult to make the argument that you believed you had authorized access.
UncleMeat 7y ago
A friend of mine from grad school was very involved in legal issues related to cfaa stuff. According to him, weev really got screwed because he failed "the punk test", which discouraged lawyers from wanting to use him as a test case.
- hitekker 7y ago
  Curious question: what is "the punk test"?
  - jakear 7y ago
    “Would this person’s attitude make them more unpleasant to work with than others I could be representing in my already limited time”, or something to that effect. Someone who spends their free time hacking university printers to distribute white supremacist propaganda and is a proud member of the “Gay Nigger Association of America” trolling group would likely not pass.
  - throwawayjava 7y ago
    "will this person be negatively perceived by a jury of their peers, or, especially, a jury of 3 federal judge? And, will this person say dumb shit that torpedos my case for reasons other than its merits"
  - UncleMeat 7y ago
    "Is this person going to say idiotic nonsense that tanks the case because the judge or jury thinks they are just awful?" Weev may have been a good test case if he wasn't a white supremacist.
lwf 7y ago
Even if the case was tried today, 9th Cir. isn't binding on other regions of the US, and there's a bit of a split, as detailed in the opinion[1]: > In recognizing that the CFAA is best understood as an anti-intrusion statute and not as a “misappropriation statute,” we rejected the contract-based interpretation of the CFAA’s “without authorization” provision adopted by some of our sister circuits. Compare Facebook, Inc. v. Power Ventures, Inc., 844 F.3d 1058, 1067 (9th Cir. 2016), cert. denied, 138 S. Ct. 313 (2017) (“[A] violation of the terms of use of a website—without more— cannot establish liability under the CFAA.”); Nosal I, 676 F.3d at 862 (“We remain unpersuaded by the decisions of our sister circuits that interpret the CFAA broadly to cover violations of corporate computer use restrictions or violations of a duty of loyalty.”), with EF Cultural Travel BV v. Explorica, Inc., 274 F.3d 577, 583–84 (1st Cir. 2001) (holding that violations of a confidentiality agreement or other contractual restraints could give rise to a claim for unauthorized access under the CFAA); United States v. Rodriguez, 628 F.3d 1258, 1263 (11th Cir. 2010) (holding that a defendant “exceeds authorized access” when violating policies governing authorized use of databases). weev was tried in an area under the 3rd Cir. jurisdiction. Somewhat interestingly, his conviction was thrown out in 2014 on venue grounds (e.g. being tried in NJ), without addressing the statutory question.[2] [1]: pp. 27-28 [2]: https://en.wikipedia.org/wiki/Weev?oldid=912921723#cite_ref-48 https://en.wikipedia.org/wiki/Weev?oldid=912921723#cite_ref-...
- joncrane 7y ago
  Is there an AWS region in the District governed by this case, so you can just do all your web scraping from instances in that region?
devmunchies 7y ago
>except that certain entities, like the Google search engine, have express permission from LinkedIn for bot access how does this work technically? i just tried crawling a friend's profile using curl and set my user agent to Google's bot and it still was blocked.
- phs318u 7y ago
  IP range whitelisting? MASSL? EDITED: corrected auto-correct.
- dx034 7y ago
  Google describes how to verify Googlebot here: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en Most other search engine crawlers provide similar methods.

hartator 7y ago

That’s awesome news. Thanks also to the EFF for all the work they are doing to ensure fair use is still a thing. We’ll (https://serpapi.com https://serpapi.com) be donating next year.

henryfjordan 7y ago

hiQ asked the court for a preliminary injunction to stop Linkedin from denying them access, won it, and this is the result of Linkedin's appeal of that injunction. This is not the end of the case. The title is wrong. The 9th Circuit just ruled that hiQ has a decent enough argument to move forward. The question of whether them scraping a public site can violate the CFAA is not settled. > We therefore conclude that hiQ has raised a serious question as to whether the reference to access “without authorization” limits the scope of the statutory coverage to computer information for which authorization or access permission, such as password authentication, is generally required > The data hiQ seeks to access is not owned by LinkedIn and has not been demarcated by LinkedIn as private using such an authorization system. HiQ has therefore raised serious questions about whether LinkedIn may invoke the CFAA to preempt hiQ’s possibly meritorious tortious interference claim. Note the tone of the language used in the ruling. The judge makes it pretty clear that nothing is final here.

foota 7y ago
I disagree about the tone, it seems to suggest to me that the judge believes there is a strong case here for hiQ.
- henryfjordan 7y ago
  I agree, but there's a difference between "hiQ has a strong case" and "Here is the final ruling on the hiQ case". I was trying to point out that the case is not over or ruled on at all. Just the preliminary injunction.
  - deleted 7y ago
    [deleted]
  - AznHisoka 7y ago
    "Just the preliminary injunction." But I thought that was a few years ago! How many more over-rulings, or appeals do we freaking need? I really hope this is the final ruling.

payne92 7y ago

Flagged for misleading title.

akersten 7y ago
Would you mind suggesting a more accurate title?
- btown 7y ago
  9th Circuit issues injunction preventing LinkedIn from blocking hiQ's scraping of public pages (IANAL but this seems like a reasonable summary of the first paragraph of the document.)

akersten 7y ago

A good decision was reached, but it's a little worrying that the emphasis in the ruling was mostly about a weighing of business interests rather than affirming a right to access public information. If HiQ's business model had not been jeopardized by LinkedIn's business desire to block them, I fear this court could have easily gone the other way. I'd really love to see a ruling that solidifies the right of someone to access publicly available data without fear of repercussions. If this case makes it to SCOTUS, I would hope the ruling is predicated on that rather than business harm. Key paragraphs from the ruling: > In short, even if some users retain some privacy interests in their information notwithstanding their decision to make their profiles public, we cannot, on the record before us, conclude that those interests—or more specifically, LinkedIn’s interest in preventing hiQ from scraping those profiles—are significant enough to outweigh hiQ’s interest in continuing its business, which depends on accessing, analyzing, and communicating information derived from public LinkedIn profiles. > Nor do the other harms asserted by LinkedIn tip the balance of harms with regard to preliminary relief. LinkedIn invokes an interest in preventing “free riders” from using profiles posted on its platform. But LinkedIn has no protected property interest in the data contributed by its users, as the users retain ownership over their profiles. And as to the publicly available profiles, the users quite evidently intend them to be accessed by others, including for commercial purposes—for example, by employers seeking to hire individuals with certain credentials. Of course, LinkedIn could satisfy its “free rider” concern by eliminating the public access option, albeit at a cost to the preferences of many users and, possibly, to its own bottom line.

victor9000 7y ago
> If HiQ's business model had not been jeopardized I think this is more about validating hiQ's legal standing in the case.

Animats 7y ago

This action does more than that. The court left the preliminary injunction against LinkedIn in place: "The district court granted hiQ’s motion. It ordered LinkedIn to withdraw its cease-and-desist letter, to remove any existing technical barriers to hiQ’s access to public profiles, and to refrain from putting in place any legal or technical measures with the effect of blocking hiQ’s access to public profiles." So LinkedIn is prohibited from blocking hiQ's access by technical means. That's a strong holding. If this case is eventually decided in favor of hiQ, scrapers can no longer be blocked. Throttled a little, maybe, but no more than other users doing a comparable query rate.

buro9 7y ago
.
- blacksmith_tb 7y ago
  Hmm, not a lawyer, but I think if you commit one crime in the process of doing something else that's legal, you're still guilty. But LinkedIn couldn't block your spider if it came back and throttled at a reasonable rate.
- thomasjudge 7y ago
  In the opinion they specifically discuss DOS attacks: "Internet companies and the public do have a substantial interest in thwarting denial-of-service attacks and blocking abusive users, identity thieves, and other ill-intentioned actors. But we do not view the district court’s injunction as opening the door to such malicious activity"
emilfihlman 7y ago
Leaving the injunction in place is insane and a huge oversight. It amounts to making web pages carriers that cannot select who they serve. It should have said only that there is nothing judicially wrong with scraping but also not limited the rights of a service.
- dhimes 7y ago

btown 7y ago

A choice quote: > In recognizing that the CFAA is best understood as an anti-intrusion statute and not as a “misappropriation statute,” Nosal I, 676 F.3d at 857–58, we rejected the contract-based interpretation of the CFAA’s “without authorization” provision adopted by some of our sister circuits. Compare Facebook, Inc. v. Power Ventures, Inc., 844 F.3d 1058, 1067 (9th Cir. 2016), cert. denied, 138 S. Ct. 313 (2017) (“[A] violation of the terms of use of a website—without more— cannot establish liability under the CFAA.”); Nosal I, 676 F.3d at 862 (“We remain unpersuaded by the decisions of our sister circuits that interpret the CFAA broadly to cover violations of corporate computer use restrictions or violations of a duty of loyalty.”), with EF Cultural Travel BV v. Explorica, Inc., 274 F.3d 577, 583–84 (1st Cir. 2001) (holding that violations of a confidentiality agreement or other contractual restraints could give rise to a claim for unauthorized access under the CFAA); United States v. Rodriguez, 628 F.3d 1258, 1263 (11th Cir. 2010) (holding that a defendant “exceeds authorized access” when violating policies governing authorized use of databases). And: > As one prominent commentator has put it, “an authentication requirement, such as a password gate, is needed to create the necessary barrier that divides open spaces from closed spaces on the Web.” Orin S. Kerr, Norms of Computer Trespass, 116 Colum. L. Rev. 1143, 1161 (2016). Moreover, elsewhere in the statute, password fraud is cited as a means by which a computer may be accessed without authorization, see 18 U.S.C. § 1030(a)(6),12 bolstering the idea that authorization is only required for password-protected sites or sites that otherwise prevent the general public from viewing the information. My layman's (IANAL) interpretation of preliminary injunctions are that the case is far from over, and this could be overturned at any time as more deliberation is done (assuming LinkedIn wants to keep throwing money at that relatively slim possibility). But now this research has been done, and future courts have the ability to look to these references and lines of reasoning all in one place. EDIT: https://twitter.com/OrinKerr/status/1171116153948626944?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet https://twitter.com/OrinKerr/status/1171116153948626944?ref_... is an analysis by the law professor cited above: > BIG NEWS: 9th Circuit holds that scraping a public website likely does not violate the CFAA, even after website owner prohibits with a cease-and-desist letter; language strongly suggests CFAA only applies to bypassing authentication. Blog post up soon. http://cdn.ca9.uscourts.gov/datastore/opinions/2019/09/09/17-16783.pdf http://cdn.ca9.uscourts.gov/datastore/opinions/2019/09/09/17... #N

7y ago

dx87 7y ago

Would that ruling mean that sites could no longer refuse to show content based on how they're accessed? For example, sites that won't load if the browser is in headless mode, or sites that depend on javascript as a way of blocking wget/curl.

freehunter 7y ago
I have a scraper for a site that used to offer an API for their publicly available site but removed the API with no warning. The info is still available to the general public, but only through their website. I created a scraper for the public page, but shortly after they switched to loading some public information through Javascript so my HTML scraper couldn't see it anymore. I ended up having to write an application around Selenium to load the Javascript and import this public information. I'm just waiting for them to start randomizing the CSS classes to make scraping even harder. The content is static, even as data changes on the server it does not refresh on the page unless you reload the page. There is no reason why your page should refuse to load plain text without Javascript enabled.
- Pfhreak 7y ago
  > There is no reason why your page should refuse to load plain text without Javascript enabled. Sure there is. You prefer writing javascript and you want to serve your site through a CDN. You might not think that's a good reason, but that's certainly a reason.
  - Franciscouzo 7y ago
    What does a CDN has to do with it?
    - Spivak 7y ago
      Because if your application bundle is a fixed asset — like a JS SPA that fetches it’s data from an API then you can distribute your entire application via an inexpensive CDN. As soon as your application bundle is rendered on your servers dynamically then only part of your site can be delivered via CDN. Basically going all-JS gives you an app model where your sever side code doesn’t even know or care about HTML or the web or whatever. It just pushes JSON or whatever around and it largely client independent. Great model when you need to support iOS, Android, web, desktop.

kevin_b_er 7y ago

Best hope that hiQ prevails. The slope slips very fast without LinkedIn's defeat. If LinkedIn prevails, the "EULA" has the force of criminal law and not just an agreement that lacks the meeting of the minds.

meowface 7y ago

Considering the kind of private scraping and selling tactics LinkedIn has been chronically guilty of (and not just the ordinary "growth hack" stuff: "LinkedIn violated data protection by using 18M email addresses of non-members to buy targeted ads on Facebook" [1]), it's satisfying to see LinkedIn lose this. [1] https://techcrunch.com/2018/11/24/linkedin-ireland-data-protection/ https://techcrunch.com/2018/11/24/linkedin-ireland-data-prot...

penagwin 7y ago
I feel like this is a really common theme I've seen several times. Something like "Music Lyric site X sues Google for embedding their lyrics in the results directly" which is funny because site X got the lyrics by scraping them from other sites. Plus Google only exists from scraping content, but I believe their TOS includes "don't scrape our content". I find it really funny that the scrapers are battling scrapers - like guys you only exist because you do THE EXACT SAME THING
- dmix 7y ago
  Regardless, there is legitimate value in the collection, cleaning, interlinking, and presentation of existing data. How that is interpreted by the law is one thing but merely because the data came from a variety of other public/private sources doesn't mean it derived all of its value externally.
  - meowface 7y ago
    For sure, but they shouldn't be hypocritical about it. If they don't consider themselves content parasites, they shouldn't consider people scraping their site to be content parasites, either. (Some sites really are just parasites, though.)
    - dodobirdlord 7y ago
      There's nothing hypocritical about it. Googlebot respects robots.txt configured on pages it scrapes. Google in turn expects that their own robots.txt will be respected. What's the issue? https://www.google.com/robots.txt https://www.google.com/robots.txt

meowface 7y ago

codedokode 7y ago

This is actually bad, would not it be better if sites would be allowed to block crawlers? I don't see what is the legal basis for forbidding to ban scrapers. Is there a law that a site must serve pages for anyone?

CamperBob2 7y ago
Parent was downvoted but I think they have a point. This sounds overbroad, to an extent that I'd worry will get the whole ruling tossed out by SCOTUS. On the other hand, if the ruling stands, it sounds like it will finally be possible to do useful things with Craigslist.
anortef 7y ago
Why? if your data is publicly accessible what is the difference between using scripts and someone hiring a ton of people on a third world country to copy&paste your content?
cooljacob204 7y ago
Because they are exposing their website to the general public. If you don't want the public to have access and be able to scrap it don't make it public facing.
3xblah 7y ago
There is no legal basis for "forbidding to ban scrapers". The question is whether there is any legal basis for banning scrapers, i.e., for blocking hiQ. In other words, if hiQ keeps scraping, are they violating anyone's rights and/or breaking the law by doing that? As long as that remains a legitimate, open question, then hiQ can argue they should be allowed to keep scraping without incurring civil or criminal liability. That is the purpose of the injunction. There could be no legal basis for blocking hiQ. Until that question is resolved, hiQ can keep on scraping.
xxxpupugo 7y ago
You can redefine what is 'public', or require login to see those information anyway. If anything, the ruling could just push websites to hide information even deeper.

CosmicShadow 7y ago

Even if LinkedIn loses and scrapers can no longer be blocked, they still just switched to putting all profiles behind an authwall, or at least it's very hard to not get an authwall. So could HiQ even carry on if they won anyway?

jolmg 7y ago
I'm not very familiar with neither LinkedIn nor HiQ, but what would be the problem with logging in before scraping?
- henryfjordan 7y ago
  The reason the pages are public to begin with is that Google will only scrape public pages for search indexing. LinkedIn wants to provide the pages ONLY to google, so they tried telling hiQ to stop scraping without any physical blockers (so as to not impede google's scraper). If LinkedIn loses this case they (and others) might try to get Google to change their policy (either use auth or some whitelisted IP addresses or something).

inbland 7y ago

What if hiQ were to then go and sell the information that's been scraped from public profiles? Would this be considered illegal?

colechristensen 7y ago
I would assume the data would still be covered by copyright meaning they could use that data and maybe create and sell derivative works, but not just scrape and publish.
- icebraining 7y ago
  Pictures and such are covered by copyright, but mere facts are not (at least in the US): https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._Rural_Telephone_Service_Co https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
  - baroffoos 7y ago
    Collections of facts are copyrightable. Which is why you can go out and make a map of your local area but you may not copy the data from google maps. You may end up with the exact same data and that is ok because you both copied the same facts but if there is a mistake on google maps (Perhaps placed as a trap) then you can be caught if your map has the same mistake.
    - icebraining 7y ago
      Only the creative aspects of those collections of facts are copyrightable. You can definitively copy from Google Maps, just like Feist could copy from Rural; what you can't copy are the aspects of the maps that require creativity. As for trap streets, they don't help as much as you think; as we've seen, simply showing that copying occurred is not enough. There was a theory that the trap streets - since they were invented, not facts - could themselves be copyrighted, but in Alexandria Drafting Co. v. Amsterdam, the courts said that copying a few trap streets among a bunch of facts was too minimal to be considered infringement.

mirimir 7y ago

This is a great opinion. But the 9th circuit is arguably still ~atypical.

supernova87a 7y ago

This case is so ridiculous on multiple fronts that although this procedural ruling (injunction) seems technically correct (to allow the case to proceed to actual court), it could just as well have been thrown out with no difference in or ultimate harm to the parties. First, LinkedIn makes the claim that its users have a right to privacy against scraping by such a 3rd party. That's laughable. As the court saw, their whole business model is made on people sharing their profiles broadly and mostly to the public. Secondly, HiQ claims that LinkedIn's efforts to stop it from using the data are tortious interference. That's bold -- suppose someone is taking your assets (you believe illegally) and selling them to others -- can you imagine the gall that the person taking your assets can sue you for interfering with their subsequent sale of your assets? Finally, that LinkedIn resorted to using the computer fraud and anti-terrorism statutes to make their argument is ridiculous. So much craziness to go around. I would've just tossed the case, but I guess there is the whole bit about due process... Maybe HiQ will fail anyway at the next substantive trial, but what a waste of time.

judge2020 7y ago
> suppose someone is taking your assets Except that, in the digital sense, it's only copied. They now have it, but you didn't lose your assets or money besides the <$0.001 it costs to serve each web page. > So much craziness to go around. I agree - I haven't read through the entire thing, but it looks like, instead of saying "you can't scrape", they could implicity give a license to users for personal and business use, but not be allowed the reselling of the data (of course carefully worded to allow the likes of Recruiters and whatnot to do so). It's like trying to argue that the DMCA says you can't create a torrent file of some movie.

victor9000 7y ago

On how this case relates to the CFAA: We therefore conclude that hiQ has raised a serious question as to whether the reference to access “without authorization” limits the scope of the statutory coverage to computer information for which authorization or access permission, such as password authentication, is generally required. Put differently, the CFAA contemplates the existence of three kinds of computer information: (1) information for which access is open to the general public and permission is not required, (2) information for which authorization is required and has been given, and (3) information for which authorization is required but has not been given (or, in the case of the prohibition on exceeding authorized access, has not been given for the part of the system accessed). Public LinkedIn profiles, available to anyone with an Internet connection, fall into the first category. With regard to such information, the “breaking and entering” analogue invoked so frequently during congressional consideration has no application, and the concept of “without authorization” is inapt.

cwkoss 7y ago

In tangentially related news, Reid Hoffman, LinkedIn co-founder, spent some time last week defending Ito's connections to pedophile Epstein. https://twitter.com/anandwrites/status/1169952673988300800 https://twitter.com/anandwrites/status/1169952673988300800

sebastianconcpt 7y ago

In short, even if some users retain some privacy interests in their information notwithstanding their decision to make their profiles public, we cannot, on the record before us, conclude that those interests—or more specifically, LinkedIn’s interest in preventing hiQ from scraping those profiles—are significant enough to outweigh hiQ’s interest in continuing its business, which depends on accessing, analyzing, and communicating information derived from public LinkedIn profiles. Reasonable. If a platform helps you make information of an individual public, then why it should matter for the platform how the market uses that public information?

sheeshkebab 7y ago

Great, finally. Don’t pretend the information is public and then block access to it. It’s either private and inaccessible, or it’s free for all to use.

deleted 7y ago

[deleted]

ummonk 7y ago

Does this have implications for other scraping? (E.g. https://techcrunch.com/2019/05/20/instagram-influencer-celebrity-accounts-scraped/ https://techcrunch.com/2019/05/20/instagram-influencer-celeb...)

sometinsome 7y ago
Or this other case https://gizmodo.com/instagram-boots-ad-partner-hyp3r-for-reportedly-scrapin-1837051804 https://gizmodo.com/instagram-boots-ad-partner-hyp3r-for-rep...

OrgNet 7y ago

I got 80,000 recipes from allrecipes.com... glad to know it was legal

asah 7y ago

Congrats Darren! Can't wait for your next one to come out of stealth mode.

dzonga 7y ago

how about websites using nonsense css-classes usually autogenerated through frameworks that make scrapping difficult ? I'm sure this ruling doesn't cover that case ? well globally I wish authorities would rule that public data should be published in computer accessible format e.g pdf's for humans and xblr's / csv for machines e.g in financial reports. lots of data in pdf's that costs a ton to mine. & tools like AWS Textract are hardly up to task.

cookie_monsta 7y ago

So the champion of the public internet turns out to be a company that scrapes your social media, MLs it and sells the results to your HR dept? I'm reminded of Dave Chapelle's Halle Berry routine...

anthony_doan 7y ago

Sweet, I scraped a bit of data and have been holding off on putting it on to my web app. I think I'll transform the data a bit before putting it up there.

monksy 7y ago

RIP Aaron Swartz

nirse 7y ago

I have never heard of a 'cause-and-desist' letter before. Odd spelling mistake for such a document.

thuylun2412 7y ago

nice

Shivetya 7y ago

Volokh take is an interesting read [1] I am curious how quickly most pages get put behind authorization. With the wording of this ruling you could pretty much go snap up any blog side (say like medium) and more. I wonder what kind of services would come out of that, having the data in a format it can be more easier parsed/analyzed? so every ecommerce site is fair game? I assume most are already being scraped but I cannot imagine having to be in an environment where many of your connections are not people [1] https://reason.com/2019/09/09/scraping-a-public-website-doesnt-violate-the-cfaa-ninth-circuit-mostly-holds/ https://reason.com/2019/09/09/scraping-a-public-website-does...

ankurkwv 7y ago
Hmm - I think a key in the ruling here was that LinkedIn maintains no copyright claim on these pages. Users on LinkedIn retain ownership of their profile data. Compare that to a blog and maybe copyright could come into play? Not a lawyer just thinking out loud...

sha666sum 7y ago

This is a weird case, as it turns the question of scraping on its head. Normally you'd think "am I allowed to scrape?", but instead the question becomes "am I allowed to prevent scraping?". Anyways, I disagree with the court's judgment here. The users have consented that their data be used in accordance with LinkedIn's privacy policy. Even if it is publicly posted does not mean that the user has relinquished control over their personal information for another company to do with as they wish.

prirun 7y ago

eBay had better lawyers than LinkedIn: https://casetext.com/case/ebay-v-bidders-edge https://casetext.com/case/ebay-v-bidders-edge I'm glad this court ruled it wasn't a violation of CFAA. But using trespass to prevent it seems reasonable. A private business should be allowed to restrict certain kinds of use of its resources (servers, bandwidth, etc), especially if it is beyond typical use. But if the load is typical and doesn't actually harm LinkedIn, it seems less reasonable to restrict them. If LinkedIn doesn't want automated access to their data because it is too much of a load on their servers, then they should be required to ban ALL automated access, including Google's bots. Of course they want Google's bots because that sends them traffic. Another reason I think it was stupid for LinkedIn to use CFAA is that it sets them up to be a protected computer system, with protected information. If that is the case, it seems they could be liable for disclosing the information to someone a user didn't want, like a stalked. It's rather dumb: LI is claiming they host protected information, but it is only protected against someone that might compete with them.

silentguy 7y ago
> If LinkedIn doesn't want automated access to their data because it is too much of a load on their servers, then they should be required to ban ALL automated access, including Google's bots. Of course they want Google's bots because that sends them traffic. By that logic, I should have access to linked premium features for free. Why should linkedin give more data to people who pay?

V3ritas1337 7y ago

Of course it isn't, it's known as implicit authorisation.

fulldecent2 7y ago

LinkedIn is a public website? Every time I click from DDG it shows a login page and zero content.