9 ms·

Unpopular opinion: when you make a HTTP request you're asking the server to give you information. The server has the right to say no. IMHO, LinkedIn doesn't ha

Unpopular opinion: when you make a HTTP request you're asking the server to give you information. The server has the right to say no. IMHO, LinkedIn doesn't have a right to stop scraping after the fact, but they have the right to take technical steps to stop scrapers from accessing their site.

problems 9y ago
I completely agree with this, but if a determined company is scraping the data, they can do various things to make the traffic blend in, it's not always possible for technical means to detect scraping. To which I say "oh well, deal with it or put it behind authentication", but some may disagree.
OkGoDoIt 9y ago
LinkedIn takes plenty of technical precautions to block scraping. I’ve built bots that scrape them in the past, it’s surprisingly difficult as LinkedIn is very good at determining you’re a bot and blocking you. So it’s hard to argue that a service which is scraping LinkedIn is doing it without knowledge that they are going against LinkedIn’s wishes. Whether or not this is illegal is up to the courts to determine, and I really hope they decide it is fine (although I have zero faith that’s how the ruling will come down). That being said, I hate LinkedIn as a company and I fully support anyone trying to mess with them. They are not a social network, they are a sleazy website that convinces people to willingly provide personal information which they then turn around and sell at ridiculously high prices. Even if you are legitimately using LinkedIn as an end-user, it’s easy to get blocked for using it too much and being forced to pay just to interact with people on the site.
- DeusExMachina 9y ago
  * That being said, I hate LinkedIn as a company and I fully support anyone trying to mess with them So it's fine to mess with them, even illegally, just because you don't like them? * convinces people to willingly provide personal information Convincing is not forcing, and in fact, you say "willingly" yourself. Any business convinces you to willingly give them money or other value. * sell at ridiculously high prices To respect to what? Prices are determined by the market. We are not talking about a life-saving medicine or health care, on which there could be some debate. We are talking about a company selling information, which has value and which it acquires through an infrastructure that takes a lot of money to run. Value that customers are willing to pay for. * Even if you are legitimately using LinkedIn as an end-user, it’s easy to get blocked For what definition of legitimately? Yours? Since it's their business, they can define what is a legitimate free use and what can be a paid one. * They are not a social network So based on what I pointed out, they are not a social network only because they are not free? Do all social network have to be free and give your data for advertisers to be social networks?
  - leksak 9y ago
    This might be one of the best replies on HN that I have seen. Spot on.
  - vkou 9y ago
    > So it's fine to mess with them, even illegally, just because you don't like them? If it's not illegal, then scummy behaviour against a scummy company doesn't exactly set my moral compass off. You reap what you sow.
    - komali2 9y ago
      >moral compass The issue I see with this is in 2 part. First, any issue that comes down to "moral compass" is inherently dangerous. We can find many examples of the simple concept that what to one person is Good is to another Evil. In this case, I think Linkedin shareholders would not appreciate calls to mess with the site, or with people having trouble jobhunting because the site is going down repeatedly due to DDOS or whatever. The second is that these kinds of calls to action (linkedin sucks, fuck with it) smell like vigilantism to me, and while Batman is my favorite hero (I really only lift because I kinda sorta wanna be batman), vigilantism doesn't contribute to a stable society. Rule of law works better than the chaos of multiple agents enforcing their own moral code as law. EDIT: I'm happy to be downvoted if I'm saying something stupid, but while doing so I would very much appreciate a quick comment as to why I'm wrong so I can improve my knowledge.
      - sprafa 9y ago
        everything comes down to a moral compass of some kind. Your comment expects some sort of objective measurement of good and evil, but I don't see any. The law can be, and is often, in the wrong. Most people seem to think this law (if it is held up in court) is wrong and should be changed. You might say, oh well, we have a democratic right to change or influence our laws. But a princeton study has found no correlation between public preferences of the majority of the population and enacted policy: https://scholar.princeton.edu/sites/default/files/mgilens/files/gilens_and_page_2014_-testing_theories_of_american_politics.doc.pdf https://scholar.princeton.edu/sites/default/files/mgilens/fi... From memory, the only thing that leads populations to revolt against their government is high enough food prices. Outside of that, revolts almost never happen. What I'm trying to say is A) unjust, monopolistic, or excessive laws are probably more normal than the opposite because B) the idea that democracy means people have some weigh in lawmaking might be a myth and C) most people don't do anything about it because they only act when the very basics of their livelihood are threatened. Your view, that some unjust or excessive laws are preferable to total chaos, seems to carry the assumption that laws are naturally benign and/or made to serve some purpose for society, therefore we should not challenge them without good reasons to do so. If the opposite is true and most laws or a high enough number of them are not just, then the fact that the vast majority of people disagrees with them is only natural. This is a fairly long-winded way of saying that most people would say you're being downvoted because there are plenty of terrible laws that we should not acquiesce to silently.
- lgas 9y ago
  > So it’s hard to argue that a service which is scraping LinkedIn is doing it without knowledge that they are going against LinkedIn’s wishes. As far as I can tell, no one has made that argument, so I'm not sure why you feel the need to rebut it. I think it all pretty much boils down to this quote from the article: > LinkedIn's position disturbs Orin Kerr, a legal scholar at George Washington University. "You can't publish to the world and then say 'no, you can't look at it,'" Kerr told Ars.
  - betenoire 9y ago
    > no one has made that argument, so I'm not sure why you feel the need to rebut it. The title is "It’s illegal to scrape our website without permission". So that argument is implied in the headline, at least. As for Orin Kerr, I'm sure he'd agree that there are private parts of the internet (my payment information being an obvious example). Just because something is deployed to the internet doesn't mean it is "published to the world" as he claims.
    - derefr 9y ago
      "Published to the world" is a bit off; I'd compare it more to recording telephone conversations with corporations' public service numbers. They always tell you that you are being recorded (for "training and quality purposes") so they give up their right to not be recorded in turn. If a company paid a million people to call into an information service hotline and each request one fact from it—and then the company recorded and compiled the answers into their own database to start their own information service—is that illegal?
      - KekDemaga 9y ago
        >If a company paid a million people to call into an information service hotline and each request one fact from it—and then the company recorded and compiled the answers into their own database to start their own information service—is that illegal? I sure hope not. From there it doesn't seem to far off to claim that "He was really only ever showing up for work to learn some skills and how he's using those skills to run his own business!" sort of lawsuits. If you compile difficult to find but freely available information into a more easy to digest format I see that as virtually always a net positive.
- jonbarker 9y ago
  Don't the other social networks throttle http requests from certain dubious ip addresses? I think they all do this.
- vertex-four 9y ago
  > a sleazy website that convinces people to willingly provide personal information which they then turn around and sell at ridiculously high prices I think you just defined a social network, unfortunately.
jfoutz 9y ago
I mostly agree. the traditional method [1] should be honored. but clearly, there are some bad agents out there that ignore the rules and mess things up for everyone. They are pretty clear about their expectations: # Notice: The use of robots or other automated means to access LinkedIn without # the express permission of LinkedIn is strictly prohibited. [1] https://www.linkedin.com/robots.txt https://www.linkedin.com/robots.txt
- franze 9y ago
  puuuh, somebody should explain the concept of user-agent groups to them (would heavily simplify their robots.txt)
jasonkostempski 9y ago
I don't think that's an unpopular opinion. But thinking they can/should have the right to press charges against you for trying would be, I hope.
benchaney 9y ago
I don't think that is an unpopular opinion. It also implies that LinkedIn is wrong, because of their HTTP server says yes, they want to retroactively say no.
ChuckMcM 9y ago
I don't think you've characterized this accurately. When you make an HTTP request to LinkedIn you are accessing their service. There is a long history of this relationship, you plug your house into the sewer line and you connect to the sewer service. You connect to the power pole and connect to the electricity service. You connect to the telephone pole and connect to the telephone service. Every service has "terms of service" which are the conditions that you are allowed to access the service and what you may do with the service once being granted access. For example, if you start pouring toxic waste into your sewer, you will find that the city will both disconnect you from the service and they will fine you for violating the terms of service you nominally agreed to when being hooked up. In LinkedIn's case, they allow you to access their service, with HTTP, to render a page in a browser for viewing of that page. Full stop. Any other use of the data you acquire over HTTP, or any other method of acquiring said data over HTTP is disallowed by the terms of service. Not only does LinkedIn have a legal right to stop scraping after the fact, they have literally centuries of common law in support of that position.
- lelandbatey 9y ago
  When you make a connection to the city sewers or to the power company, there is some kind of pre-connection step where the terms are presented and you agree to those terms. With HTTP and LinkedIn, there is no such step. There's no pre-connection agreement. LinkedIn could present such an agreement on first connection, but they do not.
  - ladon86 9y ago
    You agree to the terms of service when you sign up. If you're talking about making anonymous requests to their service, they only allow a few of those before they stop showing you profiles. If you circumvent that protection, it's a bit more like hooking a cable up to a power line (illegal) or dumping your commercial waste in the sewer (illegal).
  - ChuckMcM 9y ago
    That argument has been tried in a variety of ways and been shot down in court repeatedly. (there are parallels to tenants not agreeing to the terms of their internet connection where the landlord provided it). LinkedIn has two things that they do which protect them; First, they specify they disallow access in their robots.txt file. While not a binding agreement per se it is the default mechanism that is accepted by the community for apriori identifying whether or not automated access is possible. Second, when they detect an access pattern that violates their terms of service they actively block the access proactively notify the source of the violation. The sad truth is that web scraping has been around since the very beginnings of the Web back in 1993 and this question has been litigated in every way that you might choose to argue it, the body of case law is enough to fill at least two volumes in the reference section of the library. There is no legal or ethical basis for scraping the web without permission. And if it isn't explicitly allowed by a site the presumption is that it is disallowed (no 'open door' exception).
Scirra_Tom 9y ago
Isn't the server giving you permission to view the data? Not give? EG, if it returns an image - it doesn't imply I can use the image anywhere I want.
Shorel 9y ago
Legally, I think scrappers should respect robots.txt
- ojosilva 9y ago
  They do have an amazingly large robots.txt in fact. https://www.linkedin.com/robots.txt https://www.linkedin.com/robots.txt
mgalka 9y ago
Have no doubt this will be unpopular, but I think LinkedIn is right. Most news sites publish to the world. But scraping a news site's content and monetizing it yourself is not ok. Legally, it violates intellectual property law. But laws aside, I assume most people would agree, if someone spent the time researching and writing an article, they should have the right to monetize it and nobody else. In this case, IP law may not apply, but the concept is the same. I don't love LinkedIn myself. But they spent the time building a platform for collecting that info. I don't see why it should be OK for other people to scrape and monetize it.
- revmoo 9y ago
  Content no, facts, yes.
- deleted 9y ago
  [deleted]
sgustard 9y ago
They also state this clearly in their terms of service. "You agree that you will not ... develop, support or use software, devices, scripts, robots, or any other means or processes (including crawlers, browser plugins and add-ons, or any other technology or manual work) to scrape the Services." https://www.linkedin.com/legal/user-agreement https://www.linkedin.com/legal/user-agreement That said, this should be a breach of contract issue. It's an overreach to invoke federal fraud law.
flukus 9y ago
Generally I agree, but with linkedin anything of value seems to require logging in, which means their terms of service come into play. This is very different to scrapping data that they make open and browsable by all.

problems 9y ago

I completely agree with this, but if a determined company is scraping the data, they can do various things to make the traffic blend in, it's not always possible for technical means to detect scraping. To which I say "oh well, deal with it or put it behind authentication", but some may disagree.

OkGoDoIt 9y ago

LinkedIn takes plenty of technical precautions to block scraping. I’ve built bots that scrape them in the past, it’s surprisingly difficult as LinkedIn is very good at determining you’re a bot and blocking you. So it’s hard to argue that a service which is scraping LinkedIn is doing it without knowledge that they are going against LinkedIn’s wishes. Whether or not this is illegal is up to the courts to determine, and I really hope they decide it is fine (although I have zero faith that’s how the ruling will come down). That being said, I hate LinkedIn as a company and I fully support anyone trying to mess with them. They are not a social network, they are a sleazy website that convinces people to willingly provide personal information which they then turn around and sell at ridiculously high prices. Even if you are legitimately using LinkedIn as an end-user, it’s easy to get blocked for using it too much and being forced to pay just to interact with people on the site.

DeusExMachina 9y ago
* That being said, I hate LinkedIn as a company and I fully support anyone trying to mess with them So it's fine to mess with them, even illegally, just because you don't like them? * convinces people to willingly provide personal information Convincing is not forcing, and in fact, you say "willingly" yourself. Any business convinces you to willingly give them money or other value. * sell at ridiculously high prices To respect to what? Prices are determined by the market. We are not talking about a life-saving medicine or health care, on which there could be some debate. We are talking about a company selling information, which has value and which it acquires through an infrastructure that takes a lot of money to run. Value that customers are willing to pay for. * Even if you are legitimately using LinkedIn as an end-user, it’s easy to get blocked For what definition of legitimately? Yours? Since it's their business, they can define what is a legitimate free use and what can be a paid one. * They are not a social network So based on what I pointed out, they are not a social network only because they are not free? Do all social network have to be free and give your data for advertisers to be social networks?
- leksak 9y ago
  This might be one of the best replies on HN that I have seen. Spot on.
- vkou 9y ago
  > So it's fine to mess with them, even illegally, just because you don't like them? If it's not illegal, then scummy behaviour against a scummy company doesn't exactly set my moral compass off. You reap what you sow.
  - komali2 9y ago
    >moral compass The issue I see with this is in 2 part. First, any issue that comes down to "moral compass" is inherently dangerous. We can find many examples of the simple concept that what to one person is Good is to another Evil. In this case, I think Linkedin shareholders would not appreciate calls to mess with the site, or with people having trouble jobhunting because the site is going down repeatedly due to DDOS or whatever. The second is that these kinds of calls to action (linkedin sucks, fuck with it) smell like vigilantism to me, and while Batman is my favorite hero (I really only lift because I kinda sorta wanna be batman), vigilantism doesn't contribute to a stable society. Rule of law works better than the chaos of multiple agents enforcing their own moral code as law. EDIT: I'm happy to be downvoted if I'm saying something stupid, but while doing so I would very much appreciate a quick comment as to why I'm wrong so I can improve my knowledge.
    - sprafa 9y ago
      everything comes down to a moral compass of some kind. Your comment expects some sort of objective measurement of good and evil, but I don't see any. The law can be, and is often, in the wrong. Most people seem to think this law (if it is held up in court) is wrong and should be changed. You might say, oh well, we have a democratic right to change or influence our laws. But a princeton study has found no correlation between public preferences of the majority of the population and enacted policy: https://scholar.princeton.edu/sites/default/files/mgilens/files/gilens_and_page_2014_-testing_theories_of_american_politics.doc.pdf https://scholar.princeton.edu/sites/default/files/mgilens/fi... From memory, the only thing that leads populations to revolt against their government is high enough food prices. Outside of that, revolts almost never happen. What I'm trying to say is A) unjust, monopolistic, or excessive laws are probably more normal than the opposite because B) the idea that democracy means people have some weigh in lawmaking might be a myth and C) most people don't do anything about it because they only act when the very basics of their livelihood are threatened. Your view, that some unjust or excessive laws are preferable to total chaos, seems to carry the assumption that laws are naturally benign and/or made to serve some purpose for society, therefore we should not challenge them without good reasons to do so. If the opposite is true and most laws or a high enough number of them are not just, then the fact that the vast majority of people disagrees with them is only natural. This is a fairly long-winded way of saying that most people would say you're being downvoted because there are plenty of terrible laws that we should not acquiesce to silently.
lgas 9y ago
> So it’s hard to argue that a service which is scraping LinkedIn is doing it without knowledge that they are going against LinkedIn’s wishes. As far as I can tell, no one has made that argument, so I'm not sure why you feel the need to rebut it. I think it all pretty much boils down to this quote from the article: > LinkedIn's position disturbs Orin Kerr, a legal scholar at George Washington University. "You can't publish to the world and then say 'no, you can't look at it,'" Kerr told Ars.
- betenoire 9y ago
  > no one has made that argument, so I'm not sure why you feel the need to rebut it. The title is "It’s illegal to scrape our website without permission". So that argument is implied in the headline, at least. As for Orin Kerr, I'm sure he'd agree that there are private parts of the internet (my payment information being an obvious example). Just because something is deployed to the internet doesn't mean it is "published to the world" as he claims.
  - derefr 9y ago
    "Published to the world" is a bit off; I'd compare it more to recording telephone conversations with corporations' public service numbers. They always tell you that you are being recorded (for "training and quality purposes") so they give up their right to not be recorded in turn. If a company paid a million people to call into an information service hotline and each request one fact from it—and then the company recorded and compiled the answers into their own database to start their own information service—is that illegal?
    - KekDemaga 9y ago
      >If a company paid a million people to call into an information service hotline and each request one fact from it—and then the company recorded and compiled the answers into their own database to start their own information service—is that illegal? I sure hope not. From there it doesn't seem to far off to claim that "He was really only ever showing up for work to learn some skills and how he's using those skills to run his own business!" sort of lawsuits. If you compile difficult to find but freely available information into a more easy to digest format I see that as virtually always a net positive.
jonbarker 9y ago
Don't the other social networks throttle http requests from certain dubious ip addresses? I think they all do this.
vertex-four 9y ago
> a sleazy website that convinces people to willingly provide personal information which they then turn around and sell at ridiculously high prices I think you just defined a social network, unfortunately.

jfoutz 9y ago

I mostly agree. the traditional method [1] should be honored. but clearly, there are some bad agents out there that ignore the rules and mess things up for everyone. They are pretty clear about their expectations: # Notice: The use of robots or other automated means to access LinkedIn without # the express permission of LinkedIn is strictly prohibited. [1] https://www.linkedin.com/robots.txt https://www.linkedin.com/robots.txt

franze 9y ago
puuuh, somebody should explain the concept of user-agent groups to them (would heavily simplify their robots.txt)

jasonkostempski 9y ago

I don't think that's an unpopular opinion. But thinking they can/should have the right to press charges against you for trying would be, I hope.

benchaney 9y ago

I don't think that is an unpopular opinion. It also implies that LinkedIn is wrong, because of their HTTP server says yes, they want to retroactively say no.

ChuckMcM 9y ago

I don't think you've characterized this accurately. When you make an HTTP request to LinkedIn you are accessing their service. There is a long history of this relationship, you plug your house into the sewer line and you connect to the sewer service. You connect to the power pole and connect to the electricity service. You connect to the telephone pole and connect to the telephone service. Every service has "terms of service" which are the conditions that you are allowed to access the service and what you may do with the service once being granted access. For example, if you start pouring toxic waste into your sewer, you will find that the city will both disconnect you from the service and they will fine you for violating the terms of service you nominally agreed to when being hooked up. In LinkedIn's case, they allow you to access their service, with HTTP, to render a page in a browser for viewing of that page. Full stop. Any other use of the data you acquire over HTTP, or any other method of acquiring said data over HTTP is disallowed by the terms of service. Not only does LinkedIn have a legal right to stop scraping after the fact, they have literally centuries of common law in support of that position.

lelandbatey 9y ago
When you make a connection to the city sewers or to the power company, there is some kind of pre-connection step where the terms are presented and you agree to those terms. With HTTP and LinkedIn, there is no such step. There's no pre-connection agreement. LinkedIn could present such an agreement on first connection, but they do not.
- ladon86 9y ago
  You agree to the terms of service when you sign up. If you're talking about making anonymous requests to their service, they only allow a few of those before they stop showing you profiles. If you circumvent that protection, it's a bit more like hooking a cable up to a power line (illegal) or dumping your commercial waste in the sewer (illegal).
- ChuckMcM 9y ago
  That argument has been tried in a variety of ways and been shot down in court repeatedly. (there are parallels to tenants not agreeing to the terms of their internet connection where the landlord provided it). LinkedIn has two things that they do which protect them; First, they specify they disallow access in their robots.txt file. While not a binding agreement per se it is the default mechanism that is accepted by the community for apriori identifying whether or not automated access is possible. Second, when they detect an access pattern that violates their terms of service they actively block the access proactively notify the source of the violation. The sad truth is that web scraping has been around since the very beginnings of the Web back in 1993 and this question has been litigated in every way that you might choose to argue it, the body of case law is enough to fill at least two volumes in the reference section of the library. There is no legal or ethical basis for scraping the web without permission. And if it isn't explicitly allowed by a site the presumption is that it is disallowed (no 'open door' exception).

Scirra_Tom 9y ago

Isn't the server giving you permission to view the data? Not give? EG, if it returns an image - it doesn't imply I can use the image anywhere I want.

Shorel 9y ago

Legally, I think scrappers should respect robots.txt

ojosilva 9y ago
They do have an amazingly large robots.txt in fact. https://www.linkedin.com/robots.txt https://www.linkedin.com/robots.txt

mgalka 9y ago

Have no doubt this will be unpopular, but I think LinkedIn is right. Most news sites publish to the world. But scraping a news site's content and monetizing it yourself is not ok. Legally, it violates intellectual property law. But laws aside, I assume most people would agree, if someone spent the time researching and writing an article, they should have the right to monetize it and nobody else. In this case, IP law may not apply, but the concept is the same. I don't love LinkedIn myself. But they spent the time building a platform for collecting that info. I don't see why it should be OK for other people to scrape and monetize it.

revmoo 9y ago
Content no, facts, yes.
deleted 9y ago
[deleted]

sgustard 9y ago

They also state this clearly in their terms of service. "You agree that you will not ... develop, support or use software, devices, scripts, robots, or any other means or processes (including crawlers, browser plugins and add-ons, or any other technology or manual work) to scrape the Services." https://www.linkedin.com/legal/user-agreement https://www.linkedin.com/legal/user-agreement That said, this should be a breach of contract issue. It's an overreach to invoke federal fraud law.

flukus 9y ago

Generally I agree, but with linkedin anything of value seems to require logging in, which means their terms of service come into play. This is very different to scrapping data that they make open and browsable by all.