22 ms·
LinkedIn: It’s illegal to scrape our website without permission
- pgeorgep 9y agoNot sure how to feel about this. While in theory, scraping data is a shady practice - companies like LinkedIn leave the door open for it.
- RazorMonkey 9y agoIronic because LinkedIn scrapes and collects every scrap of contact info they can find. I've got people I sent an email to once in 199? suddenly popping up suggesting we know each other.
- lightbyte 9y agoHow is scraping data shady? Is it also shady to walk into a store and look at things without buying anything?
- cr1895 9y ago>Is it also shady to walk into a store and look at things without buying anything? Not making a claim about scraping, but it's maybe more apt to compare to walking into a store and writing a list of everything for sale their costs.
- pgeorgep 9y agoIt's more like going into the store, taking all the pennies from the free penny jar, then peacing out.
- and0 9y agoScraping isn't shady.
- pgeorgep 9y ago... and neither is picking gum from under subway railings
- atomicone 9y ago"HiQ scrapes data about thousands of employees from public LinkedIn profiles, then packages the data for sale to employers worried about their employees quitting". cute.
- 0x4f3759df 9y agoI'd always been suspicious of linking up with my current employer/coworkers, now I guess it was a valid concern.
- TomSawyer 9y agoA few weeks ago I was looking up public profiles on LinkedIn and I noticed what I interpreted to be some network-side fingerprinting of some kind. The first couple profiles came up, but from that point I was only served a sign up page. It didn't matter if I changed browsers or spawned new incognito sessions.
- mgkimsal 9y agopossibly just filtering based on your IP?
- TomSawyer 9y agoThat crossed my mind, but it'd mean they'd be willing to lock out large networks behind a NAT. I was wondering if there's something more sophisticated available.
- dylz 9y ago100% they are willing to lock out NAT networks, have been subject to this more than once browsing normally.
- JoblessWonder 9y agoAlso, they are just locking out users that aren't signed in. If a user has a login, they are probably much more likely to err on the side of caution before locking down access in some way.
- jlebrech 9y agoyou can only view a few profiles at a time, and changing to incognito window will do the same.
- wccrawford 9y ago>One plausible reading of the law—the one LinkedIn is advocating—is that once a website operator asks you to stop accessing its site, you commit a crime if you don't comply. I'm fine with that. But I'm also fine if they do the like the article suggests and require anything non-scrapable to be behind an account prompt, even if everyone with a account can access it. I don't think it's fair to make Linked In foot the bill for someone else's business. They shouldn't have to serve that content to people who aren't actually their users.
- deleted 9y ago[deleted]
- mbillie1 9y ago> They shouldn't have to serve that content to people who aren't actually their users. So put it behind a password. It's not reasonable to expect to get only the benefits of publicly-searchable data without any of the drawbacks.
- Spivak 9y agoThis is the crux of the issue, it's my impression that the CFAA doesn't have any language that sites have to take precautions to prevent unauthorized access.
- mywittyname 9y agoThe policy is also harmful to innovation. If a Google-like competitor started, all Google would have to do to crush them is demand big-name sites formally prohibit the competitor from accessing their content or risk being delisted. And magically, it becomes impossible/illegal to build a duck-duck-go.
- ue_ 9y agoIsn't that monopolistic behaviour? I suspect that it might violate some laws, though I don't know anything in particular. On the other hand, why couldn't DDG just do what Google does and say "robots.txt prevents us from getting a description for this site"? On one hand I'm against the cartel beahviour of Google doing something like that, but on the other hand, if Google asks and the other company agrees to block DDG, why shouldn't that be allowed?
- Tharkun 9y ago"illegal"? Under what law?
- haikuginger 9y agoThe CFAA, which makes it a crime to access a computer system without proper permission.
- Tharkun 9y agoThat's a load of controversial hogwash. If you have permission to view the page, then you have permission to scrape the page. Anyone can make a LinkedIn account, ergo anyone can scrape any public profile.
- pavel_lishin 9y agoJust because a law is stupid, or one you disagree with, doesn't make it any less of a law.
- tyingq 9y agoPeople have gone to jail for it. https://www.wired.com/2010/11/wiseguys-plead-guilty/ https://www.wired.com/2010/11/wiseguys-plead-guilty/
- c3833174 9y agoI'd say "buy up all tickets with fake data for scalping" weighted in more than just passive scraping.
- tyingq 9y agoHard to say...they were charged with both wire fraud and "hacking" (CFAA). The EFF said this, so they seem to agree that it set a bad precedent, even for those just scraping. "Under the government's theory, anyone who disregards – or doesn't read – the terms of service on any website could face computer crime charges," said EFF civil liberties director Jennifer Granick in a press release at the time. "Price-comparison services, social network aggregators and users who skim a few years off their ages could all be criminals if the government prevails."
- wslh 9y agoLinkedIn and Reid Hoffman have stories (don't know Reid personally) that I don't buy as one of the first LinkedIn users, premium, ads, and rejected API customer. I think LinkedIn is a scam, where the service is done for the recruiters and job postings while they make you believe they are thinking in you. They never work for an OkCupid or FB like services but for businesses except adding basic introduction messaging.
- MagnumOpus 9y ago> LinkedIn is a scam, where the service is done for the recruiters Everyone knows that LinkedIn gets paid by recruiters rather than recruitees. But of course, that is perfectly fine. People are happy to be the product if it results in job leads.
- hashkb 9y agoI think LI get paid by ads.
- wslh 9y agoThis is not what they are selling with the premium offering and this why they are lying. They are offering a way to improve selling your products and services but without any technical innovation beyond messaging.
- pinaceae 9y agoLinkedIn has two main functions: 1., Self-updating rolodex for sales people. 2., Recruitment tool. It works for both, no real competitor in sight due to the massive network effect. Some niche networks are doing ok, like Xing in DACH region. Not aware of anything special in China, guess everyone is on WeChat anyhow.
- wslh 9y agoThis is not what they are offering, look at the premium offerings: https://premium.linkedin.com/ https://premium.linkedin.com/ do you know the ROI of using LinkedIn Premium (e.g. InMail) to just contact people using a zillion of methods available, and then adding them to your LinkedIn...
- dsacco 9y agoI'm following this story with a lot of interest. I've done (and still do!) a lot of data crawling/scraping. In the past I've worked on so-called "alternative data" collection and analysis for financial forecasting. Without going into too much detail, a lot of hedge funds have teams constantly searching for kernels of data that can contribute some kind of signal for market movements. This data can come in the form of satellite imagery for oil tankers or manufacturing centers, but it can also come from the very creative use of scraped and aggregated data. It's typically very difficult to identify, collect and analyze on a technical level (as 'chollida1 has lamented in the past: normalization, labeling/bucketing and analysis of disparate data across different formats, sources and processing timeframes is a pernicious problem at this scale). From a compliance standpoint there are also generally strict requirements governing legality of use. Depending on the specific data, you might be capable of predicting earnings or broader market movements with a <5% margin of error each quarter for years at a time (I've personally seen and worked on projects with <1%, but that's the exception, not the norm). That tactic is usually found at discretionary funds; at quantitative funds the uses are much more abstract and cross-pollinated so as not to target single-equities, but rather holistic trends. Regardless, every fund is using data in some way these days; it's just a matter of how sophisticated, creative abstract they get in their analysis of it. hiQ Labs doesn't collect data for this specific purpose, but it is absolutely related. In the past I have stayed away from crawling LinkedIn and Yelp precisely because they are very litigious (regardless of the eventual outcome and legality). Now that there's another relatively high profile case out in the open like this, I'm interested in seeing how it proceeds and what the ramifications will be for companies that collect data across a wide range of uses. As Grimmelman mentioned in the article, this can impact a lot of types of businesses, not just those in the same space as hiQ. Outside of finance I am familiar with many tech companies which (openly or otherwise), kickstarted what are now widely known enterprises through cleverly crawling or scraping massive amounts of data.
- hellbanner 9y ago"satellite imagery for oil tankers" - Interesting. If hedge funds floating drones above oil tankers, I'd guess they'd be accussed of corporate espionage / spying / invasion of privacy? Ok, so oil tankers are big and "in the clear". What if $TANKERCORP floats big parachute balloons above its tankers to imply "looking past these is unauthorized viewing"? Then if a HedgeFund gets a clever angle on a satellite photo.. is that the equivalent of breaking a lock, or violating CFAA?
- jalfresi 9y agoIf you post it on a public network, it is defacto, public. If you don't want it scraped, take it down, or put it behind a login. If the user provides the login to a scraper, then the scraper has permission.
- dsacco 9y agoThat's a pretty literalist, one-size fits all approach to policy. I don't think it's a good framework to use for applying ethics considerations. If I can walk near a pool, should I also be able to run? Is running anything more than faster walking? If I'm allowed to be around the pool walking with my entry ID, should I also be allowed to place my ID on a little motorized car and make it dart around the pool really fast? Should I be able to duplicate my badge, put it on a bunch of little cars and direct them to quickly get all the floaties before anyone else? How about giving them all their own fake IDs? Now all the same questions, except there is a sign that explicitly prohibits all of these examples except for walking. It seems disingenuous to argue that the automation and rapid increase of a thing should be allowed just because a thing is allowed. That doesn't typically match our intuitive notions of ethics in other parts of society, like driving or walking around a pool. Yes, you can walk around a pool as much as you want, but if you change to running then you have fundamentally altered your behavior through increased capability, not merely done "more of walking" to utilize more of your freedom. I suppose a natural counterargument to this analogy might be that running around pools is unsafe, and scraping is not unsafe in the same way. But my point here is establishing that a behavior intrinsically changes into a different behavior if you increase the speed at which you're doing it or the capability at which you can do it.
- type0 9y ago> I don't think it's a good framework to use for applying ethics considerations. What does is it have to do with ethics? CFAA law does not apply here and that is what matters.
- Spivak 9y agoI'm interested in this point. I was under the impression that the CFAA would apply here because they've explicitly stated what users are authorized and not authorized to do on their platform. Other than incur shame from the tech community they don't have to actually prevent the unauthorized access or use.
- hyperion2010 9y agoAh, complaining when people look at the painting in your store front gallery on 5th avenue because you have to pay for the upkeep of the sidewalk.
- deleted 9y ago[deleted]
- huhtenberg 9y agoActually, no. It's more like complaining about others selling tickets to view said painting from the sidewalk. HiQ repackages and sells data it scrapes from LinkedIn.
- williamle8300 9y agoActually, no. It's like taking pictures of the paintings from the street, and reselling those pictures. If they don't like that... that the painting down. Simple.
- magnetic 9y agoSounds like an Intellectual Property violation to me. You don't have to hide your IP in order to protect it: there are laws that deal with this.
- hyperion2010 9y agoAh, complaining when people look at the painting in your store front gallery on 5th avenue because you have to pay for the upkeep of the sidewalk.
- snarfy 9y agoThat law needs to be abused into the ground. We should all be filing frivolous lawsuits claiming CFAA violations whenever someone we don't like is accessing our sites. All we need is a little disclaimer in 5pt text stating who does not have authorization, like say all members of Congress.
- yourapostasy 9y agoEven if successfully litigated, doesn't this just move the scraping activity to less-obvious means, and to better-funded scrapers less concerned with legalities, making LinkedIn's efforts to clamp down upon scraping even more difficult? Is there a business case for LinkedIn to monetize the scraping by selling access to an API instead? Between bot-nets, mechanical turks, deep learning, data brokering, lack of globally-enforced privacy laws that require divulging of sourced personal data, etc., I can't see a way for LinkedIn to prevent others from scraping and gaining from their publicly- and user-accessible data. They'll drive it underground, but if the concern is preventing others from grabbing the data at all, versus performance management, they'll still leak like a sieve.
- whatnotests 9y agoLinkedIn gutted their API a couple years ago and now the information they took out has been moved to a private API If your business is in the recruiter space, expect to have your API keys revoked and to receive a cease and desist letter as well.
- yourapostasy 9y agoThis sounds like the setup to an up-spiraling arms race with "dark scrapers". I'm guessing behind-the-scenes, LinkedIn figures they can out-spend the extralegal scrapers, and likely considers their efforts will deliver halo effects to the rest of Microsoft. It would be educational to hear how LinkedIn plans to take down bot-net-based scraping that uses deep learning to identify patterns that successfully mimic human users and bypass their bot detection; could possibly help other white hats who want to battle bots and general malware.
- whatnotests 9y agoIf the goal of such an operation were to effectively create an alternative to LinkedIn, along the likes of other "claim your listing" sites, then this could be a worthwhile cause.
- kharms 9y ago
- hellbanner 9y agoRelated: (also frontpage): https://news.ycombinator.com/item?id=14891192 https://news.ycombinator.com/item?id=14891192
- FedericoCapell0 9y ago""The CFAA makes it a crime to "access a computer without authorization or exceed authorized access." Courts have been struggling to figure out what this means ever since Congress passed it more than 30 years ago."" The law by itself is ok, but I suspect lawmakers were referring to accessing a single personal workstation, probably not taking into account a cluster of servers containing public accessible data.
- mywittyname 9y ago"Authorization" should be clarified to mean requiring credentials that formally grant access. In my eyes, public-facing content on a website is explicitly available to anyone. It clearly does not require authorization.
- mattm 9y agoA perfect example of corporate hypocrisy. Both Facebook and LinkedIn did illegal things when they were younger but now that they are more established, they don't want anyone to do the same thing to them.
- graphitezepp 9y agoI mean that's just the standard way non altruistic entities interact with the law isn't it? Appeal to it when it benefits them, ignore it when it doesn't.
- deleted 9y ago[deleted]
- cartercole 9y agothey can go eat a dick... you cant stop the signal!
- jlebrech 9y agoinstead of making it a legal matter they could just start giving bogus information to the scrapers. They have no legal obligation to provide accurate information to scrapers.
- deleted 9y ago[deleted]
- aantix 9y agoHere's a few twists that would be interesting to see how they held up in court... * What if someone had a few oDesk workers to manually enter in the LinkedIn data? * What if someone wrote a scraper that parses the Google cache version of the public LinkedIn page?
- notfromhere 9y agoThe oDesk thing is allowed, I believe. Or at least can't be detected
- deleted 9y ago[deleted]
- deedubaya 9y agoLinkedIn doesn't want YOUR work history easily shareable or consumed by other services. They sell this data to the highest bidders -- typically recruiters.
- debacle 9y agoI'm seeing a lot of bad analogies thrown around in this comment thread, mostly based on emotional response and/or a dislike for LinkedIn. As someone who has done a lot of scraping in the past (sometimes for good, sometimes not), the number one thing you need to respect as a scraper is that email or phone call you get saying "Stop doing that." In almost all instances, you're legally fine in the real world until you get some communication to stop and/or blacklisted. After that point, what you are doing becomes a crime. - LinkedIn is not a public resource, it is a private company that pays for servers. - LinkedIn might scrape too, but that argument isn't going to hold up in court, and the scraping they do is probably in line with their EULA (protip: never install a social networking app on your phone, ever). - The analogy to the storefront, taking pictures in public, etc, all break down because scraping LinkedIn requires you to access their resources. - The analogy to browsing a store is great. If you are in a store, and they ask you to leave, and you don't, that's trespassing. Trespassing isn't legal. The CFAA isn't a great law. There are a lot of gray areas. But LinkedIn seems within their rights here. If anyone wants to know how this is going to wind up: https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-settle-suit-craigslist-over-use-real-estate-facts https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...
- mywittyname 9y ago> But LinkedIn seems within their rights here. Congress writes bad laws all the time. So LinkedIn might be within their rights, but that doesn't mean they should have those rights. It's bad for innovation to allow for selective discrimination like this. LinkedIn is perfectly happy to allow Google, Yahoo, Bing, and many, many more companies to scrape their content and use it for personal profit. Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic. Letting established tech giants get away with this will slowly erode American dominance in technology.
- debacle 9y ago> Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic. But it's not. LinkedIn's data is their entire business. They are within their rights to restrict access to it. This is the classic ant and grasshopper story. If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.
- deleted 9y ago[deleted]
- cdevs 9y agoI've always believed that if you are scrapable then be prepared to be scraped. The internet is still a digital Wild West and stoping these guys will not stop some underground source with a cloud flare or tor hidden service or outside your countries laws from doing what they want with what they can get. they can build a site or they can sell it, stoping these guys just stops a public and honest company, sure they can stop the public companies but the best thing to do is protect yourself. Don't leave the keys to the door out front and get mad when someone uses the keys.
- emodendroket 9y agoSince it would criminalize tons of current behavior and set a terrible precedent if this prevails, I expect them to be completely successful.
- fiatjaf 9y agoI once tried to scrape http://www.diseasesdatabase.com http://www.diseasesdatabase.com by all means, but failed so hard I can't even tell this story without embarrassment. They have all kinds of blocks in place.
- anonymous344 9y agoWeb service is not like ad on a paper or a public billboard. If service is being scraped by other companies who makes or not makes money for it, or prevents the owner making money with it (filtering out the ads) this service is most likely to cause extra costs as bandwith and processing power. They are not free. So you should have right to block anybody from your site as you wish!
- Fej 9y agoLinkedIn is free to block the IPs of the scrapers.
- keithpeter 9y ago"It's a fight that could determine whether an anti-hacking law can be used to curtail the use of scraping tools across the Web." In the US. Not elsewhere. Is it possible that the centre of gravity of innovation will be somewhere, er, less litigious soon? Reminds me of the City of London's historical approach to most forms of regulation (think: Francis Drake, privateers, the convertibility of lute strings &c).
- deleted 9y ago[deleted]
- didibus 9y agoWith LinkedIn's current policy, are they the owner of my data? What if I scrape my own profile?
- mehh 9y agoQuite likely yes they are the owners of your data.
- brunock222 9y agoI do a lot of crawler services as well. The last one, I did 13 crawlers to keeping comparing prices of all drugstore products online. I'm selling it to drugstore ecommerces who wants to know when their competitor prices and when they're doing promotions, if competitors prices are higher... Well, I just automated the job of a person that was doing this job manually every single day, looking into competitor's site If it is illegal, you have to ask to google maps to remove all houses from your database and just let in the houses who gave the permission to it. The both sentence means the same bullshit, they are both wrong: Google Maps: "The front-door of my house is faced to a public street. That does not mean you can take photos of it on Street View and use a very smart OCR, to read my house number. Not to mention that sometimes you give my house photo to others by Captcha asking the house number, c'mon!" Linked: "My CV is half-public. That does not mean you can take crawler of it." ... This is only a cool discussion in North Korea or maybe in China. Not in the rest of the world.
- groby_b 9y agoI would suggest occasionally actually informing yourself of the wider world. Your very example of Google Maps in Germany - well, turns out Google had to give people the option to opt out: https://europe.googleblog.com/2010/10/how-many-german-households-have-opted.html https://europe.googleblog.com/2010/10/how-many-german-househ... Same goes for web sites - robots.txt exists for a reason. If your crawler ignores that, well, I'd suggest talking to a lawyer.
- briandear 9y agoI wasn’t aware that robots.txt had any legal meaning. Am I wrong?
- jasonkostempski 9y agoIf you are wrong then there is no sanity left in the world.
- pdabbadabba 9y ago> Am I wrong? I hate to be the bearer of bad new but...maybe. A reading of the Computer Fraud and Abuse Act could make robots.txt legally enforceable. And given the government's approach to CFAA cases a very aggressive interpretation, under the right circumstances (for example, when it provides evidence that the scraper knew that scrapint was not authorized), seems like a real possibility. Among the many other things CFAA criminalizes, it makes it a crime to "intentionally access[] a computer without authorization or exceed[] authorized access . . and thereby obtain[] information from any protected computer;" A "protected computer" is, among other things, a computer "which is used in or affecting interstate or foreign commerce or communication." That would probably cover just about any server on the Internet. https://www.law.cornell.edu/uscode/text/18/1030 https://www.law.cornell.edu/uscode/text/18/1030
- athenot 9y agoMeanwhile LinkedIn was happy to scrape its users' email archives and contact lists. It would appear there's some karma at play here...
- eadz 9y agoThat is an interesting fact.. LinkedIn broke the ruling against scraping they are relying on to prevent scraping.
- throwaway91111 9y agoScraping would be forbidden via TOS; theoretically it would be the users giving access to linked in (who actually is bound by the terms) who would be liable, not linked in.
- pmiller2 9y agoThat still wouldn't prevent an entity (person or corporation) that never agreed to the TOS from scraping.
- bb611 9y agoIf it's illegal to scrape without permission, that makes the behavior of scraping illegal. They are not claiming it's a TOS violation, they're claiming it's a CFAA violation, which is a federal statue and (theoretically) applies equally to everyone.
- throwaway91111 9y agoMy point is that someone with permission gave them access.
- bb611 9y agoHaving "permission" is not an applicable defense to a federal criminal charge, the law supersedes some random person (i.e. a LinkedIn user) saying "oh yeah, that's okay."
- rocky1138 9y agoAs far as I'm concerned, if the information is publicly visible on a site (not behind a login) and as long as the scraping doesn't cause performance issues or generate costs on the site, then it should be perfectly fine to do.
- pmlnr 9y ago> As far as I'm concerned, if the information is publicly visible on a site then it should be perfectly fine to do. There, I fixed it.
- rocky1138 9y agoI don't think it's okay to cause enough stress to the web host to cause delays for other visitors.
- mehh 9y agoWhilst that is your opinion, it is not a fact, nor is it legally valid.
- 5trokerac3 9y agoUnpopular opinion: when you make a HTTP request you're asking the server to give you information. The server has the right to say no. IMHO, LinkedIn doesn't have a right to stop scraping after the fact, but they have the right to take technical steps to stop scrapers from accessing their site.
- problems 9y agoI completely agree with this, but if a determined company is scraping the data, they can do various things to make the traffic blend in, it's not always possible for technical means to detect scraping. To which I say "oh well, deal with it or put it behind authentication", but some may disagree.
- OkGoDoIt 9y agoLinkedIn takes plenty of technical precautions to block scraping. I’ve built bots that scrape them in the past, it’s surprisingly difficult as LinkedIn is very good at determining you’re a bot and blocking you. So it’s hard to argue that a service which is scraping LinkedIn is doing it without knowledge that they are going against LinkedIn’s wishes. Whether or not this is illegal is up to the courts to determine, and I really hope they decide it is fine (although I have zero faith that’s how the ruling will come down). That being said, I hate LinkedIn as a company and I fully support anyone trying to mess with them. They are not a social network, they are a sleazy website that convinces people to willingly provide personal information which they then turn around and sell at ridiculously high prices. Even if you are legitimately using LinkedIn as an end-user, it’s easy to get blocked for using it too much and being forced to pay just to interact with people on the site.
- DeusExMachina 9y ago* That being said, I hate LinkedIn as a company and I fully support anyone trying to mess with them So it's fine to mess with them, even illegally, just because you don't like them? * convinces people to willingly provide personal information Convincing is not forcing, and in fact, you say "willingly" yourself. Any business convinces you to willingly give them money or other value. * sell at ridiculously high prices To respect to what? Prices are determined by the market. We are not talking about a life-saving medicine or health care, on which there could be some debate. We are talking about a company selling information, which has value and which it acquires through an infrastructure that takes a lot of money to run. Value that customers are willing to pay for. * Even if you are legitimately using LinkedIn as an end-user, it’s easy to get blocked For what definition of legitimately? Yours? Since it's their business, they can define what is a legitimate free use and what can be a paid one. * They are not a social network So based on what I pointed out, they are not a social network only because they are not free? Do all social network have to be free and give your data for advertisers to be social networks?
- ijafri 9y agoI guess for LinkedIn it could matter most, because their only livelihood is CV and employer companies Data, if you steal a poor man's bread they are gonna be upset. It was dead as a social network long ago if it ever was, now their only asset is the data, so I guess their right to protect their only asset is legit.
- int_19h 9y agoTheir desire to protect it is legit, but it doesn't necessarily give them a right to do so. A long time ago, companies that were compiling phonebooks also thought that they could use e.g. copyright to prevent others from using that compiled data for commercial gain. They were wrong, because copyright doesn't protect "mere aggregation" of data, as courts have ruled. In this case, LinkedIn is not arguing on the basis of copyright, so it's a different legal argument. But the essence of the case is they same - they want to have a business model around aggregating other people's data and then providing access to that data, while limiting what people who access it can do with it. They don't have a right to this business model. If technical means of restricting access don't work, or if adopting them means that they drive most of their customers away, tough luck.
- ijafri 9y agoYes it's a consensus user-generarated content or facts are not liable to copy rights, I am not sure about the legality however it should not take long to prove as such. I meant there is nothing that LinkedIn 'created' out of that users data. But it's little complex or little immoral to steal users data hosted on their infrastructure that costs them a lot.
- int_19h 9y agoIn US, it's an established legal fact. https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._Rural_Telephone_Service_Co https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R.... (Note, HN strips out the trailing dot from the URL above.)
- artur_makly 9y agoit is incredible how many global companies are successfully scraping LI as part of their core business. you would be suprised
- pmlnr 9y agoWhen I make my CV public on LinkedIn I expect it to be public. Even for bots. That is the whole point of 'public.'
- funkymike 9y agoLinkedIn's purpose is not to help you as a worker. It's purpose is to scrape together as much personal information about people as possible to make money from it. Likewise Facebook doesn't exist to help connect people together to create a more personal, connected world. It exists to connect people so that they can get the users to share as much personal information as possible, so they can profit from that information. It's always important to remember that on social media you are the product. You have to weigh what you benefits it is really providing vs. what you are giving up.
- smegel 9y agoI can't even view LinkedIn with a web browser, how the hell do robots get in?
- twsted 9y agoWhat if I build a new public search engine and LinkedIn blocks me because I am not Google or Bing? They could stop potential (and much needed) competitors.
- bostonvaulter2 9y agoIt would be interesting to make a "search engine" that returns results that can be aggregated. In the LinkedIn case that would be over a company.
- rebootthesystem 9y agoIt would be very good to have one or a few solid legal decisions relating to scraping. Today, as far as I know, the entire segment is still in limbo. It is important to understand that, legal or not, in the US anyone can sue you for anything. In another comment people were discussing the legality of following or ignoring "robots.txt". I tend to be pragmatic about this stuff. If you fabricate your own legal interpretation and end-up being sued by LinkedIn, it could end-up costing you $250K. When facing large corporations law firms often ask for sizable retainers ($100K+) and proof of cash-on-hand to go beyond that, $250K total not being an unusual number. They don't do this out of greed. They do it because litigation at those levels can be very expensive. If you only have $100K you could find yourself burning through all of it quickly. If you don't have more cash to continue you'll lose the lawsuit and the $100K you spent will be burned for no reason at all. In other words, the law firm is protecting you by asking you to have enough cash to litigate. A few years ago we started to develop a very extensive product based on obtaining data from Amazon. Some of the data is available through their API and other data had to be scraped in various forms. The product is extremely valuable yet the issues pertaining to scraping made me decide to put it on hold. Even if you can make millions a year the prospect facing a monster like Amazon in a lawsuit, as improbable as it might be, is scary enough to go look for other pastures.
- adrr 9y agoIt would be interesting to put a TOS into your scraper by the pragma or useragent field that says if you return data to the scraper you accept the TOS.
- dna_polymerase 9y agoOh you don't want your stuff scraped? Well then get the fuck of the public internet. Simple as that.
- point78 9y agoI'd have to agree with you. Public info is public info.
- danschumann 9y agoFrom a non-legal perspective, this is a battle of employers vs employees rights. Do employers have the right to (in a sense) spy on their employees, watching their every move to see if they want to switch jobs?
- tomxor 9y agoHypocrites... They should have lost any legal ownership to information freely available on their site long ago when they decided it was ok to scrape private information from their users email accounts without authorisation.
- duncan_bayne 9y agoYet another reason to shut down your LinkedIn account and walk away.
- osullip 9y agoDoes this case mean, if LinkedIn wins, I can write a cease and desist to Facebook, LinkedIn and Google to stop 'accessing' my PC with cookies and tracking my data?
- jug 9y agoIt is illegal to let your browser fetch this text without my permission.
- mehh 9y agoI suspect we don't need your permission as under the terms of use of this sight you forfeit your rights to this user generated content, and the rights now belong to Hacker News.
- nilved 9y agoIf I spammed people as much as LinkedIn I wouldn't be throwing stones. They completely ignore the law.
- cerf 9y agoSometimes I wish we could start over.
- euske 9y agoBack in the days I wrote my own crawler and layout analyzer to collect news articles for my research (cf. http://www.unixuser.org/~euske/python/webstemmer/ http://www.unixuser.org/~euske/python/webstemmer/ ). I thought in 2005 it was mostly acceptable as long as you don't hog their bandwidth, but today I feel it would be looked differently. It is kinda sad to see that the more and more part of the web is treated not public. It seems that everyone likes to build their wall.
- WalterBright 9y agoI seem to recall that the courts ruled that a phone book is copyrightable, but the data within it is not. Perhaps that is precedent?
- bschwindHN 9y ago> HiQ scrapes data about thousands of employees from public LinkedIn profiles, then packages the data for sale to employers worried about their employees quitting Screw both of these companies. One runs a shitty "professional" social network (data collection tool) and the other scoops up their data droppings and makes it their core business. I just can't have sympathy for either side.
- troisx 9y agoOnce again the 9th circuit makes a ridiculous decision that hopefully gets overturned. The fact that they sided with Facebook in that lawsuit shows how out-of-touch they are with reality. Hopefully it gets overturned.
- drefgert 9y agoIs there ANY data that can be read from LinkedIn programritically without their permission?
- drefgert 9y agoSeems to me LinkedIn is missing the chance to be a source of identity verification for other services.
- jamesmattis 9y agoIs scraping public data on a specific platform like LinkedIn is illegal? Why? According to TechCrunch, they sued hundreds of "Anonymous" individuals last year. How can unnamed people can be sued?
- honestoHeminway 9y agoLinkedIns Email Password ConArtistery has inspired a whole generation of Anti-Malware and Lawyer-Plugins, preventing the layman from giving away his data, even on "friendly" sites. Im going to turn around now, and whatever happens to this site is going to happen. They worked so hard, to ask for this.
- toniprada 9y ago> To expand its user base, Power asked users to provide their Facebook credentials and then—with their permission—sent Power.com invitations to their Facebook friends. Facebook, naturally, didn't appreciate this marketing tactic. They sent Power a cease-and-desist letter and also blocked the IP addresses Power was using to communicate with Facebook's servers. > Facebook sued, claiming that its cease-and-desist letter made Power's access unauthorized under the terms of the CFAA. Power disagreed and argued that having permission from Facebook users was good enough—it didn't need separate approval from Facebook itself. How can be illegal if users are giving their permission? What happens if I give my permission to an external service to extract my own data?
- a_lifters_life 9y agoWhat a joke, as if LinkedIn is some important website more so than <name a million others>