4 ms·

I'm seeing a lot of bad analogies thrown around in this comment thread, mostly based on emotional response and/or a dislike for LinkedIn. As someone who has do

I'm seeing a lot of bad analogies thrown around in this comment thread, mostly based on emotional response and/or a dislike for LinkedIn. As someone who has done a lot of scraping in the past (sometimes for good, sometimes not), the number one thing you need to respect as a scraper is that email or phone call you get saying "Stop doing that." In almost all instances, you're legally fine in the real world until you get some communication to stop and/or blacklisted. After that point, what you are doing becomes a crime. - LinkedIn is not a public resource, it is a private company that pays for servers. - LinkedIn might scrape too, but that argument isn't going to hold up in court, and the scraping they do is probably in line with their EULA (protip: never install a social networking app on your phone, ever). - The analogy to the storefront, taking pictures in public, etc, all break down because scraping LinkedIn requires you to access their resources. - The analogy to browsing a store is great. If you are in a store, and they ask you to leave, and you don't, that's trespassing. Trespassing isn't legal. The CFAA isn't a great law. There are a lot of gray areas. But LinkedIn seems within their rights here. If anyone wants to know how this is going to wind up: https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-settle-suit-craigslist-over-use-real-estate-facts https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...

mywittyname 9y ago
> But LinkedIn seems within their rights here. Congress writes bad laws all the time. So LinkedIn might be within their rights, but that doesn't mean they should have those rights. It's bad for innovation to allow for selective discrimination like this. LinkedIn is perfectly happy to allow Google, Yahoo, Bing, and many, many more companies to scrape their content and use it for personal profit. Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic. Letting established tech giants get away with this will slowly erode American dominance in technology.
- debacle 9y ago
  > Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic. But it's not. LinkedIn's data is their entire business. They are within their rights to restrict access to it. This is the classic ant and grasshopper story. If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.
  - mywittyname 9y ago
    They are within their right to require authorization to their content, yes, but they don't do that. They make it public and allow some companies to scrape that content and resell it for profit (like Google), but while restricting other companies from doing the same. If they don't want their data to be public, then they shouldn't make it public. They could require authorization to view any content on the site and solve this problem instantly.
    - debacle 9y ago
      What leads you to believe that LinkedIn's data is public?
      - int_handler 9y ago
        I believe mywittyname is specifically referring to the LinkedIn profile pages that are publicly visible without requiring a login and is not claiming that all LinkedIn's data is public. Thus, the question is why doesn't LinkedIn simply hide all profile data behind user authentication?
  - int_19h 9y ago
    > But it's not. LinkedIn's data is their entire business. "There has grown in the minds of certain groups in this country the idea that just because a man or corporation has made a profit out of the public for a number of years, the government and the courts are charged with guaranteeing such a profit in the future, even in the face of changing circumstances and contrary to public interest. This strange doctrine is supported by neither statute or common law. Neither corporations or individuals have the right to come into court and ask that the clock of history be stopped, or turned back." > If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves. https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._Rural_Telephone_Service_Co https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
- flashman 9y ago
  The question is, how would you better define hacking? To make the letter of the law match the spirit? "Exceeding authorised access" is an early attempt that predates modern internet usage. You can't just say "anything the computer lets you do, is legal" because code exploits are just a computer following the (poorly-written) instructions in its code.
  - mywittyname 9y ago
    I'm not a lawyer, but it should a start with a clearly demonstrated attempt at prohibiting access to content. The words, "don't use this" are not enough. There needs to be active, ongoing safeguards to protect the data, i.e., authorization tokens, credentials, encryption keys, etc. In the 80s, it was reasonable to assume that connecting to some port on a remote machine owned by another person or company could constitute unauthorized access. But today, billions of people connect to ports on remote machines thousands of times a day for completely legitimate reasons, so it's reasonable to assume that data that can be accessed by just asking nicely over the internet is considered intended for public consumption. It seems permissive, but I think that's a crucial component. If some company makes accidentally makes their S3 buckets public, it's completely unfair to say that accessing that information is illegal, especially when they are serving up other information in public S3 buckets which they want people to access.
kstrauser 9y ago
Many years ago, I wrote a scraper for a certain .mil financial website. My company at the time - and its customers - had legitimate accounts on that website, but the site was horrible to use. Page loads took upwards of a minute, and there were no front-page notification systems to let you know if any of the information on the site had changed. Kind of a big deal when it was a window into the government's accounts payable system. For instance, if you didn't notice that an auditor had left a comment on an invoice, and you didn't respond to it within X hours, then the gov't could deny the invoice and you'd have to start over. The end result was our customers paying full-time employees to manually examine every single open invoice in the .mil system every single day, which just increased load on the website and made it even slower. Enter my scraper. It copied data into a local PostgreSQL database that our customers could run reports against. A process that used to take a human 6 hours a day now took 30 error-free seconds. The scraper was even a benefit to the website as we ran it overnight during low-load conditions, and because my software was smarter than a web browser, we could retrieve the same information as a human with about 1/3 the number of web requests. Perfect, right! Well, no. We got an angry call from the developers complaining that we were the ones making their site slow, even though 1) we measurably didn't, 2) we were lighter on resources than the humans we were replacing, and 3) we scrupulously obeyed their rate limiting requirements and erred on the side of running 10% slower than they had originally requested from their customers. That particular problem went away when we pointed out that their shiny new website wasn't Section 508 accessibility compliant and could not be made to be without literally throwing out their entire web service and starting over, but our website was, and that if they continued to allow us to serve screen reader compatible pages to our disabled customers then there wouldn't be a need to have the .mil website shut down and a Congressional investigation launched. All parties involved decided this was a reasonable compromise.

mywittyname 9y ago

> But LinkedIn seems within their rights here. Congress writes bad laws all the time. So LinkedIn might be within their rights, but that doesn't mean they should have those rights. It's bad for innovation to allow for selective discrimination like this. LinkedIn is perfectly happy to allow Google, Yahoo, Bing, and many, many more companies to scrape their content and use it for personal profit. Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic. Letting established tech giants get away with this will slowly erode American dominance in technology.

debacle 9y ago
> Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic. But it's not. LinkedIn's data is their entire business. They are within their rights to restrict access to it. This is the classic ant and grasshopper story. If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.
- mywittyname 9y ago
  They are within their right to require authorization to their content, yes, but they don't do that. They make it public and allow some companies to scrape that content and resell it for profit (like Google), but while restricting other companies from doing the same. If they don't want their data to be public, then they shouldn't make it public. They could require authorization to view any content on the site and solve this problem instantly.
  - debacle 9y ago
    What leads you to believe that LinkedIn's data is public?
    - int_handler 9y ago
      I believe mywittyname is specifically referring to the LinkedIn profile pages that are publicly visible without requiring a login and is not claiming that all LinkedIn's data is public. Thus, the question is why doesn't LinkedIn simply hide all profile data behind user authentication?
- int_19h 9y ago
  > But it's not. LinkedIn's data is their entire business. "There has grown in the minds of certain groups in this country the idea that just because a man or corporation has made a profit out of the public for a number of years, the government and the courts are charged with guaranteeing such a profit in the future, even in the face of changing circumstances and contrary to public interest. This strange doctrine is supported by neither statute or common law. Neither corporations or individuals have the right to come into court and ask that the clock of history be stopped, or turned back." > If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves. https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._Rural_Telephone_Service_Co https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
flashman 9y ago
The question is, how would you better define hacking? To make the letter of the law match the spirit? "Exceeding authorised access" is an early attempt that predates modern internet usage. You can't just say "anything the computer lets you do, is legal" because code exploits are just a computer following the (poorly-written) instructions in its code.
- mywittyname 9y ago
  I'm not a lawyer, but it should a start with a clearly demonstrated attempt at prohibiting access to content. The words, "don't use this" are not enough. There needs to be active, ongoing safeguards to protect the data, i.e., authorization tokens, credentials, encryption keys, etc. In the 80s, it was reasonable to assume that connecting to some port on a remote machine owned by another person or company could constitute unauthorized access. But today, billions of people connect to ports on remote machines thousands of times a day for completely legitimate reasons, so it's reasonable to assume that data that can be accessed by just asking nicely over the internet is considered intended for public consumption. It seems permissive, but I think that's a crucial component. If some company makes accidentally makes their S3 buckets public, it's completely unfair to say that accessing that information is illegal, especially when they are serving up other information in public S3 buckets which they want people to access.

kstrauser 9y ago

Many years ago, I wrote a scraper for a certain .mil financial website. My company at the time - and its customers - had legitimate accounts on that website, but the site was horrible to use. Page loads took upwards of a minute, and there were no front-page notification systems to let you know if any of the information on the site had changed. Kind of a big deal when it was a window into the government's accounts payable system. For instance, if you didn't notice that an auditor had left a comment on an invoice, and you didn't respond to it within X hours, then the gov't could deny the invoice and you'd have to start over. The end result was our customers paying full-time employees to manually examine every single open invoice in the .mil system every single day, which just increased load on the website and made it even slower. Enter my scraper. It copied data into a local PostgreSQL database that our customers could run reports against. A process that used to take a human 6 hours a day now took 30 error-free seconds. The scraper was even a benefit to the website as we ran it overnight during low-load conditions, and because my software was smarter than a web browser, we could retrieve the same information as a human with about 1/3 the number of web requests. Perfect, right! Well, no. We got an angry call from the developers complaining that we were the ones making their site slow, even though 1) we measurably didn't, 2) we were lighter on resources than the humans we were replacing, and 3) we scrupulously obeyed their rate limiting requirements and erred on the side of running 10% slower than they had originally requested from their customers. That particular problem went away when we pointed out that their shiny new website wasn't Section 508 accessibility compliant and could not be made to be without literally throwing out their entire web service and starting over, but our website was, and that if they continued to allow us to serve screen reader compatible pages to our disabled customers then there wouldn't be a need to have the .mil website shut down and a Congressional investigation launched. All parties involved decided this was a reasonable compromise.