7 ms·

Harvesting LinkedIn data for fun and profit

216 points by stremovsky 7y ago

VeryHacker 7y ago
That creepy. Good catch OP
jszymborski 7y ago
Just a reminder that using data obtained illegal (e.g. the 2012 LinkedIn hack) is also illegal (depending on your jurisdiction, IANAL, et cetera...)
- scohesc 7y ago
  I'm pretty sure if you're using it for research purposes like the numerous other people out there and you can prove it's not malicious, you'll be fine.
  - bureaucrat 7y ago
    Don’t keep those bombs like Capital One hacker Paige Thompson did. You might hit an extra jailtime jackpot.
- gumby 7y ago
  At least in the western United States, scraping is fine: http://cdn.ca9.uscourts.gov/datastore/opinions/2019/09/09/17-16783.pdf http://cdn.ca9.uscourts.gov/datastore/opinions/2019/09/09/17... . LinkedIn was even a party to this case! The UID->email data dump was not, AFAIK, legal though.
  - shkkmo 7y ago
    That is not at all what that ruling says. The ruling just upholds an injunction until the case is decided. It explicitly does not pupport to provide any precedent on the legality of scraping.
    - calny 7y ago
      That's not exactly right. You're correct the ruling only upheld a preliminary injunction. Like you imply, that's a provisional remedy before trial, subject to change. But in practice the court is unlikely to change its views about how the CFAA operates. And rulings about preliminary injunctions are frequently cited as precedent. Here, the opinion strongly suggested the CFAA does not prohibit scraping of publicly available data: "It is likely that when a computer network generally permits public access to its data, a user’s accessing that publicly available data will not constitute access without authorization under the CFAA." (Opinion at 33.) Since this was a preliminary injunction, this passage won't be binding on other courts; however, it certainly will be cited as persuasive precedent. So will more policy-oriented passages like the following: "giving companies like LinkedIn free rein to decide, on any basis, who can collect and use data—data that the companies do not own, that they otherwise make publicly available to viewers, and that the companies themselves collect and use—risks the possible creation of information monopolies that would disserve the public interest." (Opinion at 36.)
      - shkkmo 7y ago
        I am no legal expert, but the judge explicitly warns against reading too much into him upholding the injunction: > I emphasize that appealing from a preliminary injunction to obtain an appellate court’s view of the merits often leads to “unnecessary delay to the parties and inefficient use of judicial resources.” Sports Form, 686 F.2d at 753. These appeals generally provide “little guidance” because “of the limited scope of our review of the law” and “because the fully developed factual record may be materially different from that initially before the district court.” The opinion does cite other 9th circuit decisions that imply that the 9th circuit believes that the CFAA does not prohibit scraping, but also explicitly notes that the CFAA is not the only relevant law. > We note that entities that view themselves as victims of data scraping are not without resort, even if the CFAA does not apply: state law trespass to chattels claims may still be available I don't see where my claims overstep what is laid out in that opinion. Edit: The opinion does indicate that there is a good chance that the 9th circuit will eventually rule that public scraping is not covered by the CFAA, but even if the 9th circuit Court does make that ruling, that still would not mean that scraping is legal under other laws.
- prepend 7y ago
  I don’t think that’s correct. As long as you didn’t collude in the illegal act, it’s legal in the entire US. Not sure about other countries. This came up most recently in the Manning/ Snowden leaks as while the leaks were illegal, using the info is not illegal. There was a lot of press and here’s a decent post by a law professor explaining legality, https://jonathanturley.org/2016/10/17/cnn-it-is-illegal-for-voters-to-possess-wikileaks-material/ https://jonathanturley.org/2016/10/17/cnn-it-is-illegal-for-... Data is different from stolen property in that it isn’t property. Once data is made public it is no longer proprietary so it can be used legally even if it was originally obtained illegally. This is different if you were paying for non-public stolen data, but no one here is talking about that.
radiusvector 7y ago
Where's the profit part? How did you monetize stolen data?
- skrebbel 7y ago
  It's a figure of speech.
  - scoobyyabbadoo 7y ago
    It's misleading when it's not correct.
- mfoy_ 7y ago
  By getting publicity he builds social capital, which he (presumably) can turn into real capital through his work. From his bio "...leading the evolution of startups and enterprises to achieve the highest level of security and compliance."
- giarc 7y ago
  I think he is referring to the start up he was working for. >During my work on the start-up, I developed techniques that allow me to collect and cross-reference a lot of personal data including data from LinkedIn.
bwb 7y ago
I think I missed it, but how did he get their emails? That part I didn't understand as I was hoping LI didn't expose that...
- papreclip 7y ago
  >Searching on Google, I found the database from the LinkedIn 2012 hack. Each record had a user id and an email without additional information. >The link to the LinkedIn user profile was missing and personal information was lacking. As a result, it was not very useful. I think he is downplaying the value of that hacked database. Without it what would he have? userid and profile url combos...
  - bwb 7y ago
    Ah right, just seems silly then, as he is relying on a hacked DB for the most private part.
  - Avamander 7y ago
    You can also enumerate users based on phone numbers, you don't need the database in that case, 10k numbers per account, probably also somehow resettable but I haven't spent that much time on it because LinkedIn didn't find it an issue.
dlphn___xyz 7y ago
this seems pretty useless...
- giarc 7y ago
  He took two data sources (LinkedIn and LinkedIn data hack) and combined them to get first, last, email, linkedin profile ID. Imagine a spammer having millions of active email addresses with first/last.
  - shkkmo 7y ago
    > Imagine a spammer having millions of active email addresses with first/last. Don't they already? There have been SOOO many breaches in this area that I rather doubt there are many active emails that don't have some publicly available dataset linking them to first and last names. The valuable thing here is linking that data to the LinkedIn profile ID.
gumby 7y ago
I don't see the problem with collecting public profiles. They are, you know, public, and people entered their own data in the interest of propagating it. nonpublic profiles (or nonpublic data from public profiles) from, say, FB would be different.
- chmars 7y ago
  Just four letters: GDPR. And anyway: Just because something is published only does mean that you can collect, store etc. it however you like. That is pretty basic in my opinion.
  - banachtarski 7y ago
    What does GDPR have anything to do with the parent post.
    - GordonS 7y ago
      The parent post explicitly says they don't have a problem with 3rd parties storing personal information hoovered up from LinkedIn. The GDPR is very relevant here - restrictions around the storage and processing of personal information is the whole point of it. While this might seem like a grey area (given the information is public), the GDPR is actually very clear here - you cannot store and process PI without the consent of those individuals.
      - hash872 7y ago
        Not in any way shape or form defending 'Harvesting LinkedIn data'- I think it's quite bad. But I'm getting a little concerned by all these 'but GPDR!' arguments I'm seeing out there. The EU is not a world government, and the GPDR should not apply to non-EU citizens. Europe cannot regulate what I do here in America- the law is not simply applicable. (And I say this as someone that supports more tech company regulation here in the US!) Things like France trying to apply your 'right to be forgotten' to the entire world's Google search results are extremely troubling. Don't apply your country/region's laws to non-citizens, please :)
bwblabs 7y ago
I use to (ab)use their Outlook Social Connector (OSC) from the now gone API (https://outlook.linkedinlabs.com/osc/people/details https://outlook.linkedinlabs.com/osc/people/details), they stopped it in 2015. I used it just to get names and profile images for easy onboarding (like gravatar). There was a LSC-Signature header that was a sha1_hmac("POST%2Fosc%2Fpeople%2Fdetails$auth_token$unix_timestamp", "aa15bd5f089eb93a5b2b4a0e11443cb78e44f34d"); which I reversed from the Social Connector DLL, I never found it posted online, but others must have done the same.
Domenic_S 7y ago
> Get rid of duodecimal profile ids. Obscurity is not a solution here. I don't think it's meant to be a security element, but to disambiguate same name collisions, right?
- applecrazy 7y ago
  This is true. The profile URL is customizable, and the profile id can be removed from the url.
datavirtue 7y ago
This is not private data. I want people to find me on LinkedIn. LOL Can't you create a bot from this and harvest all of my work history? I hope so.
downandout 7y ago
This is clickbait and does not belong on the front page of HN. The author says he scraped public profile URLs and names. When you make your profile on social websites public, then you have chosen to...make them public. He then claims he has emails from LinkedIn, but those emails are from an old data breach, and he even admits that the emails are limited to those found in a 2012 data breach. Finally, the title of this article says he did this for “fun and profit”. By the author’s own admission, the “profit” part is missing here. He claims the company “...went out of business without getting funding”. So in other words, he has access to the main LinkedIn website, found a link to a database from an old data breach, and used to work for a now defunct company. None of that translates to “Harvesting LinkedIn data for fun and profit”.
zawerf 7y ago
This is a pretty tame use of the 2012 Linkedin breach. The breach also contained unsalted hashes which has mostly been cracked by now. They all ended up in a huge collection (773 million records) containing email/password pairs from many different sources: https://www.troyhunt.com/the-773-million-record-collection-1-data-reach/ https://www.troyhunt.com/the-773-million-record-collection-1... With so many password variations for a user, you can do credential stuffing to crawl all the private accounts of an email to build a pretty complete profile of the person (not just correlate some linkedin profile like in this post). I am sure someone out there is already doing this for profit.
piqufoh 7y ago
> For the past 15 years I’ve been leading the evolution of startups and enterprises to achieve the highest level of security and compliance. ... serving up over unsecured http
tomquirk 7y ago
Here's a friendly Python library that is ideal for this: https://github.com/tomquirk/linkedin-api https://github.com/tomquirk/linkedin-api
trackofalljades 7y ago
I don't really understand the author's claim that... https://il.linkedin.com/directory/people-a-1/ https://il.linkedin.com/directory/people-a-1/ ...contains links to all public LinkedIn profiles. I looked for a bunch of people I know with public profiles and they weren't in there (and neither was I).
- stremovsky 7y ago
  This specific subdomain lists people in Israel (IL.linkedin.com). There are other subdomains for other countries.
stremovsky 7y ago
Hi, People started to look themselves in the LinkedIn index. It is country-based. I updated the article with more examples. For example: https://il.linkedin.com/directory/people-a-1/ https://il.linkedin.com/directory/people-a-1/ https://www.linkedin.com/directory/people-a-1/ https://www.linkedin.com/directory/people-a-1/ https://uk.linkedin.com/directory/people-a-1/ https://uk.linkedin.com/directory/people-a-1/ https://de.linkedin.com/directory/people-a-1/ https://de.linkedin.com/directory/people-a-1/ https://fr.linkedin.com/directory/people-a-1/ https://fr.linkedin.com/directory/people-a-1/

216 points by stremovsky 7y ago

That creepy. Good catch OP

Just a reminder that using data obtained illegal (e.g. the 2012 LinkedIn hack) is also illegal (depending on your jurisdiction, IANAL, et cetera...)

scohesc 7y ago
I'm pretty sure if you're using it for research purposes like the numerous other people out there and you can prove it's not malicious, you'll be fine.
- bureaucrat 7y ago
  Don’t keep those bombs like Capital One hacker Paige Thompson did. You might hit an extra jailtime jackpot.
gumby 7y ago
At least in the western United States, scraping is fine: http://cdn.ca9.uscourts.gov/datastore/opinions/2019/09/09/17-16783.pdf http://cdn.ca9.uscourts.gov/datastore/opinions/2019/09/09/17... . LinkedIn was even a party to this case! The UID->email data dump was not, AFAIK, legal though.
- shkkmo 7y ago
  That is not at all what that ruling says. The ruling just upholds an injunction until the case is decided. It explicitly does not pupport to provide any precedent on the legality of scraping.
  - calny 7y ago
    That's not exactly right. You're correct the ruling only upheld a preliminary injunction. Like you imply, that's a provisional remedy before trial, subject to change. But in practice the court is unlikely to change its views about how the CFAA operates. And rulings about preliminary injunctions are frequently cited as precedent. Here, the opinion strongly suggested the CFAA does not prohibit scraping of publicly available data: "It is likely that when a computer network generally permits public access to its data, a user’s accessing that publicly available data will not constitute access without authorization under the CFAA." (Opinion at 33.) Since this was a preliminary injunction, this passage won't be binding on other courts; however, it certainly will be cited as persuasive precedent. So will more policy-oriented passages like the following: "giving companies like LinkedIn free rein to decide, on any basis, who can collect and use data—data that the companies do not own, that they otherwise make publicly available to viewers, and that the companies themselves collect and use—risks the possible creation of information monopolies that would disserve the public interest." (Opinion at 36.)
    - shkkmo 7y ago
      I am no legal expert, but the judge explicitly warns against reading too much into him upholding the injunction: > I emphasize that appealing from a preliminary injunction to obtain an appellate court’s view of the merits often leads to “unnecessary delay to the parties and inefficient use of judicial resources.” Sports Form, 686 F.2d at 753. These appeals generally provide “little guidance” because “of the limited scope of our review of the law” and “because the fully developed factual record may be materially different from that initially before the district court.” The opinion does cite other 9th circuit decisions that imply that the 9th circuit believes that the CFAA does not prohibit scraping, but also explicitly notes that the CFAA is not the only relevant law. > We note that entities that view themselves as victims of data scraping are not without resort, even if the CFAA does not apply: state law trespass to chattels claims may still be available I don't see where my claims overstep what is laid out in that opinion. Edit: The opinion does indicate that there is a good chance that the 9th circuit will eventually rule that public scraping is not covered by the CFAA, but even if the 9th circuit Court does make that ruling, that still would not mean that scraping is legal under other laws.
prepend 7y ago
I don’t think that’s correct. As long as you didn’t collude in the illegal act, it’s legal in the entire US. Not sure about other countries. This came up most recently in the Manning/ Snowden leaks as while the leaks were illegal, using the info is not illegal. There was a lot of press and here’s a decent post by a law professor explaining legality, https://jonathanturley.org/2016/10/17/cnn-it-is-illegal-for-voters-to-possess-wikileaks-material/ https://jonathanturley.org/2016/10/17/cnn-it-is-illegal-for-... Data is different from stolen property in that it isn’t property. Once data is made public it is no longer proprietary so it can be used legally even if it was originally obtained illegally. This is different if you were paying for non-public stolen data, but no one here is talking about that.

radiusvector 7y ago

Where's the profit part? How did you monetize stolen data?

skrebbel 7y ago
It's a figure of speech.
- scoobyyabbadoo 7y ago
  It's misleading when it's not correct.
mfoy_ 7y ago
By getting publicity he builds social capital, which he (presumably) can turn into real capital through his work. From his bio "...leading the evolution of startups and enterprises to achieve the highest level of security and compliance."
giarc 7y ago
I think he is referring to the start up he was working for. >During my work on the start-up, I developed techniques that allow me to collect and cross-reference a lot of personal data including data from LinkedIn.

bwb 7y ago

I think I missed it, but how did he get their emails? That part I didn't understand as I was hoping LI didn't expose that...

papreclip 7y ago
>Searching on Google, I found the database from the LinkedIn 2012 hack. Each record had a user id and an email without additional information. >The link to the LinkedIn user profile was missing and personal information was lacking. As a result, it was not very useful. I think he is downplaying the value of that hacked database. Without it what would he have? userid and profile url combos...
- bwb 7y ago
  Ah right, just seems silly then, as he is relying on a hacked DB for the most private part.
- Avamander 7y ago
  You can also enumerate users based on phone numbers, you don't need the database in that case, 10k numbers per account, probably also somehow resettable but I haven't spent that much time on it because LinkedIn didn't find it an issue.

dlphn___xyz 7y ago

this seems pretty useless...

giarc 7y ago
He took two data sources (LinkedIn and LinkedIn data hack) and combined them to get first, last, email, linkedin profile ID. Imagine a spammer having millions of active email addresses with first/last.
- shkkmo 7y ago
  > Imagine a spammer having millions of active email addresses with first/last. Don't they already? There have been SOOO many breaches in this area that I rather doubt there are many active emails that don't have some publicly available dataset linking them to first and last names. The valuable thing here is linking that data to the LinkedIn profile ID.

gumby 7y ago

I don't see the problem with collecting public profiles. They are, you know, public, and people entered their own data in the interest of propagating it. nonpublic profiles (or nonpublic data from public profiles) from, say, FB would be different.

chmars 7y ago
Just four letters: GDPR. And anyway: Just because something is published only does mean that you can collect, store etc. it however you like. That is pretty basic in my opinion.
- banachtarski 7y ago
  What does GDPR have anything to do with the parent post.
  - GordonS 7y ago
    The parent post explicitly says they don't have a problem with 3rd parties storing personal information hoovered up from LinkedIn. The GDPR is very relevant here - restrictions around the storage and processing of personal information is the whole point of it. While this might seem like a grey area (given the information is public), the GDPR is actually very clear here - you cannot store and process PI without the consent of those individuals.
    - hash872 7y ago
      Not in any way shape or form defending 'Harvesting LinkedIn data'- I think it's quite bad. But I'm getting a little concerned by all these 'but GPDR!' arguments I'm seeing out there. The EU is not a world government, and the GPDR should not apply to non-EU citizens. Europe cannot regulate what I do here in America- the law is not simply applicable. (And I say this as someone that supports more tech company regulation here in the US!) Things like France trying to apply your 'right to be forgotten' to the entire world's Google search results are extremely troubling. Don't apply your country/region's laws to non-citizens, please :)

bwblabs 7y ago

I use to (ab)use their Outlook Social Connector (OSC) from the now gone API (https://outlook.linkedinlabs.com/osc/people/details https://outlook.linkedinlabs.com/osc/people/details), they stopped it in 2015. I used it just to get names and profile images for easy onboarding (like gravatar). There was a LSC-Signature header that was a sha1_hmac("POST%2Fosc%2Fpeople%2Fdetails$auth_token$unix_timestamp", "aa15bd5f089eb93a5b2b4a0e11443cb78e44f34d"); which I reversed from the Social Connector DLL, I never found it posted online, but others must have done the same.

Domenic_S 7y ago

> Get rid of duodecimal profile ids. Obscurity is not a solution here. I don't think it's meant to be a security element, but to disambiguate same name collisions, right?

applecrazy 7y ago
This is true. The profile URL is customizable, and the profile id can be removed from the url.

datavirtue 7y ago

This is not private data. I want people to find me on LinkedIn. LOL Can't you create a bot from this and harvest all of my work history? I hope so.

downandout 7y ago

This is clickbait and does not belong on the front page of HN. The author says he scraped public profile URLs and names. When you make your profile on social websites public, then you have chosen to...make them public. He then claims he has emails from LinkedIn, but those emails are from an old data breach, and he even admits that the emails are limited to those found in a 2012 data breach. Finally, the title of this article says he did this for “fun and profit”. By the author’s own admission, the “profit” part is missing here. He claims the company “...went out of business without getting funding”. So in other words, he has access to the main LinkedIn website, found a link to a database from an old data breach, and used to work for a now defunct company. None of that translates to “Harvesting LinkedIn data for fun and profit”.

zawerf 7y ago

This is a pretty tame use of the 2012 Linkedin breach. The breach also contained unsalted hashes which has mostly been cracked by now. They all ended up in a huge collection (773 million records) containing email/password pairs from many different sources: https://www.troyhunt.com/the-773-million-record-collection-1-data-reach/ https://www.troyhunt.com/the-773-million-record-collection-1... With so many password variations for a user, you can do credential stuffing to crawl all the private accounts of an email to build a pretty complete profile of the person (not just correlate some linkedin profile like in this post). I am sure someone out there is already doing this for profit.

piqufoh 7y ago

> For the past 15 years I’ve been leading the evolution of startups and enterprises to achieve the highest level of security and compliance. ... serving up over unsecured http

tomquirk 7y ago

Here's a friendly Python library that is ideal for this: https://github.com/tomquirk/linkedin-api https://github.com/tomquirk/linkedin-api

trackofalljades 7y ago

I don't really understand the author's claim that... https://il.linkedin.com/directory/people-a-1/ https://il.linkedin.com/directory/people-a-1/ ...contains links to all public LinkedIn profiles. I looked for a bunch of people I know with public profiles and they weren't in there (and neither was I).

stremovsky 7y ago
This specific subdomain lists people in Israel (IL.linkedin.com). There are other subdomains for other countries.

stremovsky 7y ago

Hi, People started to look themselves in the LinkedIn index. It is country-based. I updated the article with more examples. For example: https://il.linkedin.com/directory/people-a-1/ https://il.linkedin.com/directory/people-a-1/ https://www.linkedin.com/directory/people-a-1/ https://www.linkedin.com/directory/people-a-1/ https://uk.linkedin.com/directory/people-a-1/ https://uk.linkedin.com/directory/people-a-1/ https://de.linkedin.com/directory/people-a-1/ https://de.linkedin.com/directory/people-a-1/ https://fr.linkedin.com/directory/people-a-1/ https://fr.linkedin.com/directory/people-a-1/