6 ms·
I don't think you've characterized this accurately. When you make an HTTP request to LinkedIn you are accessing their service. There is a long history of this r
by ChuckMcM 9y ago
I don't think you've characterized this accurately. When you make an HTTP request to LinkedIn you are accessing their service. There is a long history of this relationship, you plug your house into the sewer line and you connect to the sewer service. You connect to the power pole and connect to the electricity service. You connect to the telephone pole and connect to the telephone service.
Every service has "terms of service" which are the conditions that you are allowed to access the service and what you may do with the service once being granted access. For example, if you start pouring toxic waste into your sewer, you will find that the city will both disconnect you from the service and they will fine you for violating the terms of service you nominally agreed to when being hooked up.
In LinkedIn's case, they allow you to access their service, with HTTP, to render a page in a browser for viewing of that page. Full stop. Any other use of the data you acquire over HTTP, or any other method of acquiring said data over HTTP is disallowed by the terms of service.
Not only does LinkedIn have a legal right to stop scraping after the fact, they have literally centuries of common law in support of that position.
- lelandbatey 9y agoWhen you make a connection to the city sewers or to the power company, there is some kind of pre-connection step where the terms are presented and you agree to those terms. With HTTP and LinkedIn, there is no such step. There's no pre-connection agreement. LinkedIn could present such an agreement on first connection, but they do not.
- ladon86 9y agoYou agree to the terms of service when you sign up. If you're talking about making anonymous requests to their service, they only allow a few of those before they stop showing you profiles. If you circumvent that protection, it's a bit more like hooking a cable up to a power line (illegal) or dumping your commercial waste in the sewer (illegal).
- ChuckMcM 9y agoThat argument has been tried in a variety of ways and been shot down in court repeatedly. (there are parallels to tenants not agreeing to the terms of their internet connection where the landlord provided it). LinkedIn has two things that they do which protect them; First, they specify they disallow access in their robots.txt file. While not a binding agreement per se it is the default mechanism that is accepted by the community for apriori identifying whether or not automated access is possible. Second, when they detect an access pattern that violates their terms of service they actively block the access proactively notify the source of the violation. The sad truth is that web scraping has been around since the very beginnings of the Web back in 1993 and this question has been litigated in every way that you might choose to argue it, the body of case law is enough to fill at least two volumes in the reference section of the library. There is no legal or ethical basis for scraping the web without permission. And if it isn't explicitly allowed by a site the presumption is that it is disallowed (no 'open door' exception).
- jboy55 9y agoWhen you make your conenction to Linkedin, What user agent would you provide? One that's blank, another that's "lelandbatey bot" or one that says, "Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405" ?
- saurik 9y ago(I am not a lawyer.) As far as I understand the legal precedents involved, random terms of services for websites are not effective in this scenario as the public profiles do not require having any account or other relationship. This actually went to court, and because Zappos didn't force users to click through a terms of service to access their service, the terms of service was invalid. As for their ability to control what you do with the information: there might be a limited license on the data granted from users to LinkedIn that is not transferrable, so maybe you couldn't build a service that redistributed that information, but I don't see why obtaining and holding it would be illegal. As for the analogies to power and telephone and such, those are built on property owned by a local government and there are usually other extra laws related to them: it isn't due to some common law position that you can't mess with their stuff. Here, I am not a lawyer, but I am a government official with a particular interest in sewage; here is a link to the sewer use ordinances form our local sanitation district: pay particular attention to 2.03. http://goletawest.org/wp-content/uploads/2012/04/Ordinance-No.-60-Sewer-Use.pdf http://goletawest.org/wp-content/uploads/2012/04/Ordinance-N...
- ChuckMcM 9y agoI worked at Google for four years, an independent search engine for 5 more after that, and at IBM after it acquired said search engine for 18 months after that. Everyone of those organizations spent many thousands of dollars on legal fees over just this question and reviewed tons of case law. Every single one of them concluded that based on how the law was written and how the web worked, there is no legal way to scrape a web site without its explicit permission to do so. That won't stop people from trying of course and it was a source of constant entertainment in the ops team at Blekko at how people tried to sneak around at scraping (it can get very creative) but; it isn't legal, you can and will get banned from all access for it, and if you use the results in another product or offering you will be found liable for damages.
- sokoloff 9y ago> there is no legal way to scrape a web site without its explicit permission to do so. Google scrapes several of my sites and I've never given Google explicit permission to do so.
- pmiller2 9y agoI disagree with your analogy. To me, the key word in "HTTP request" is request. A request is something that can be granted or not.
- ChuckMcM 9y agoPerhaps more clearly would be any HTTP request that LinkedIn believes is in violation of their terms of service will be denied. It can be hard to know when the first request arrives if it is someone scraping the site or not, but once it is clear that it is someone scraping they actively deny all future requests. If they could know that the request coming in was going to be a scrape and not a page view they would preemptively deny it.
- vageli 9y agoBut what is the difference between a scrape and a page view? If a human looks at it once, after scraping, does it become a page view? Is pocket downloading content on my behalf for me to read later, a scraper? What's the difference between a scraper and an offline browser who's content a human never browses?
- jboy55 9y agoTrue, but in making the request, you will provide information on who is making that request. If you say, "I am a bot!", and they grant you permission, your request is legal. But if you say, 'I am NOT a bot', like spoofing a browser's user agent string, but you are a bot, then you are requesting access under a pretense, in order to circumvent their terms of service. Kinda feels morally wrong, and illegal.
- arcbyte 9y agoThe user agent header is not an authorization header. There is explicitly an authorization header.
- havetocharge 9y agoThere isn't a universal 'I'm a bot' setting. There are user agent conventions, but they are hardly standard. Your point works in theory, but it's not something one can just implement and be reasonably confident that they won't be scraped.
- deleted 9y ago[deleted]
- aleem 9y agoThe scenario is a bit more nuanced though and creeps into Internet freedom. - Reminds me of CraigsList vs PadMapper[1]. In that scenario I side with CL -- it was right to block PM. PM or others should not be allowed to build a new UI on top of CL because CL was the one that put in years of effort of nurturing its listings, its network, building brand equity and taking associated risks and costs. - As others have highlighted, the data is publicly accessibly and there is no agreement the scraper/crawler is bound by. The agreement is between the LinkedIn user and LinkedIn. The scraper is connected to the Internet pipe crawling the Internet freely as it wants. It's not reproducing the data anywhere so copyright should not be an issue. - What if a scraper didn't scrape LinkedIn but just the Google or Archive.org cached versions and read those instead? It would not be pressuring LinkedIn server resources in this case. - What if all of my employees allow me to scrape their LinkedIn data? Can I scrape all of their info? Can LinkedIn stop me from doing that (In the case of Facebook vs Power Ventures, the answer is that LinkedIn would be able to prevent this behaviour). - Who owns the data? Medium.com doesn't own the posts. LinkedIn doesn't own the CVs. [1]: https://news.ycombinator.com/item?id=4286325
- ChuckMcM 9y agoNow go read the 3Taps vs Craigslist cases (https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc. https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc.) to start. To be clear here, I feel like I understand your argument that facts aren't copyrightable or protectable and that how you got them is not relevant. I'm just saying the legal system doesn't see it that way, they have said so in many cases, and so far everyone who has used your argument or variations of it in court has failed to prevail.
- tarsinge 9y agoThe analogy of pouring waste is not accurate, we are looking at what is done with the service, e.g. it would be the equivalent of allowing drinking the water but not cooking with it, or using electricity for specific devices. The contract is a debit/volume, what I do with it is irrelevant, and by this analogy on the web I should be allowed to scrape if I stay in the allowed bandwidth by the website.