6 ms·
Out of curiosity how do you guys think they managed to scrape LinkedIn on such a large scale? I've been wanting to do some social graph experimentation on it (
by Havoc 7y ago
Out of curiosity how do you guys think they managed to scrape LinkedIn on such a large scale?
I've been wanting to do some social graph experimentation on it (small scale - say 1000 people near me) but concluded I probably couldn't scrape enough via raw scraping without freaking out their anti-scraping. (And API is a non-starter since that basically says everything is verboten).
- alasdair_ 7y agoI scraped 10 million records from linkedin a few years ago from a single ip by using their search function. I got a list of the top 1000 first names and top 1000 last names and wrote a script to query all combinations and scrape the results. This may or may not still work.
- Ididntdothis 7y agoYou probably have to be highly distributed. At least that’s what I did when I tried to scrape a large site some years ago. I had around 100 machines in different countries and gave each of them random pages to scrape.
- auiya 7y agoUsername checks out
- magashna 7y agoIt looks like the purpose was data enrichment, so maybe it was pieced together over time from multiple sources. My linkedin from PDL only had 1 bit of wrong info. I wasn't able to find anything on my personal email addresses which is good.
- tyri_kai_psomi 7y agoDistributed bot and scraper networks. Thousands of IPs geographically dispersed throughout the world. There is only so much you can do with rate limiting.
- StuffedParrot 7y agoProxies can also work well for cheaper than buying distributed compute.
- gdulli 7y agoThey asked about LinkedIn, where the content is gated behind a login. If it was a rate limiting problem, that would be trivial. Needing to be logged in as the same user defeats the purpose of proxying to hide your physical origin. Registering thousands of different users to use in a distributed way is hard now that they require a text message verification for new accounts.
- manigandham 7y agoPublic LinkedIn profiles (which is many of them) are open to scrapers and they lost a court case about it. https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...
- gdulli 7y agoI go to LinkedIn without being logged in and nearly always get a login gate instead of the profile. They were ordered to unblock hiQ specifically, they were not ordered to open up content to scrapers generally. They can still throttle high volume traffic and put up captchas. I think the only specific thing the court ordered was for them to unblock hiQ IP ranges.
- momokoko 7y agoScraping LinkedIn is so common you can usually hire people with years of experience in it. It is not as complicated as you might think. There are at minimum hundreds of companies that sell LinkedIn data they have scraped.
- sergiotapia 7y agoA very large distributed network of machines.
- memn0nis 7y agoHey - not related to your comment (apologies) but wanted to get in touch . You left a note on a previous post of mine about wanting to simplify FTP. I'd love to work on this project and wanted to see if you'd be willing to connect so I can understand the problem better. Feel free to email me at kunal@mightydash.com, and thanks in advance!
- fredley 7y ago'Mobile Proxies' like https://oxylabs.io/mobile-proxies https://oxylabs.io/mobile-proxies (no affiliation) allow you to use large pools of mobile or domestic IPs to scrape. It's expensive, but not prohibitively so. Once you've got a mobile IP you become incredible hard to throttle, since you're behind a mobile NAT gateway.
- sillysaurusx 7y agoSo far, the answers have contained non-technical answers like "Distributed Scraping." Well, yes, obviously. A more useful answer is: I did this once, many years ago. Back then it was a matter of hooking up PhantomJS and making sure your user string was set correctly. Since PhantomJS was – I think – essentially the same as what headless chrome is today, the server can't determine that you're running a headless browser. Now, it's not so easy nowadays to do that. There are mechanisms to detect whether the client is in headless mode. But most websites don't implement advanced detection and countermeasures. And in the ideal case, you can't really detect that someone is doing automated scraping. Imagine a VM that's literally running chrome, and the script is set up to interact with the VM using nothing but mouse movements and keyboard presses. You could even throw in some AI to the mix: record some real mouse movements and keyboard presses over time, then hook up some AI to your script such that it generates movements and keyboard presses that are impossible to distinguish from real human inputs. Such a system would be almost impossible to differentiate vs your real users. The other piece of the puzzle is user accounts. You often have to have "aged" user accounts. For example, if you tried to scrape LinkedIn using your own account, it wouldn't matter if you were using 500 IPs. They would probably notice. It's hard to counter a determined scraper.
- onlyrealcuzzo 7y agoI wrote a chrome headless framework that types using semi-realistic key presses (timing, mistakes, corrections) and does semi-realistic scrolling / swiping and clicking / tapping. It's not very hard to get something that would be too hard for almost every website beside Google and Facebook to bother with. If it's a 1 on a 0-9 scale in difficulty, most websites just don't have the resources to detect it It took me like ~3 hours to write it, but I guarantee it would take months for someone to detect it, and even then, they'd have a lot of false positives and negatives.
- zer0tonin 7y agoI think there's also a lot of bot-detection-as-a-service around here that can be used by sites smaller than Google and Facebook, like WhiteOps or IAS anti-fraud.
- Matsta 7y agoLinkedIn Sales Navigator is a paid tool which allows you to search their whole database. Then depending on how much you pay you can get all their personal details (Email address, phone number, even their address sometimes.) https://business.linkedin.com/sales-solutions/sales-navigator https://business.linkedin.com/sales-solutions/sales-navigato...
- calibas 7y agoI've always been a little confused how this works. If I got all that info for free, it's a "data leak", but if I pay to get the same detailed personal information it's... In either case my personal data is given away without my consent, but there's this implication that it's only an issue when someone doesn't pay for it.
- sbarre 7y ago> In either case my personal data is given away without my consent You gave that consent when you put your info in Linkedin in the first place, according to their ToS.
- Matsta 7y agoI think everyone is confused. Everyone just wants their slice of the pie (aka $$$).
- gdulli 7y agoYou're right, my take on this is that a company scraped a bunch of publicly available information, that people left open (consciously or not.) That's why only a subset have phone numbers. The profile URLs, emails, most people don't even try to protect those. Normally the company sells this data, but now they've given it away. It's not good this data got out because the curation has some value to spammers or whoever. But using the word "leak" here undermines the severity of a real leak where passwords and social security numbers are exposed. Data that was never meant by anyone to be open. Everyone likely has (technically) provided consent for every piece of information here being shared with partners. Buried in fine print that it wasn't really expected they'd read, of course. It's the cost of being online, and that sucks, but it seems only a leak of what had already been given out.
- iamsb 7y agoYou use a proxy botnet and route your scraping requests through that. Use something like hola proxy or crawlera for example.
- kaivi 7y agoI've crawled a popular social network on a large scale, currently doing the same for dating services as a hobby. God, wish I'd still got paid for webscraping. Here are some tricks which may or may not work today: - Have an app where user logs in through said website, then scrape their friends using this user's token. That way you get exponential leverage on the number of API calls you can make, with just a handful of users. - Call their API through ipv6, because they may not yet have a proper, ipv6 subnet-based rate limiter. - Scrape the mobile website. Even Facebook still has a non-js mobile version. This single WAP/mobile website defeats every anti-scraping measure they may have. - From a purely practical perspective, start with a baremetal transaction-isolation-less database like Cassandra/ScyllaDB. Don't rely on googling "postgres vs mongodb" or "sql vs nosql", those articles will all end in "YMMV". What you really need is massive IOPS, and a multi-node ring-based index with ScyllaDB will achieve that easily. Or just use MongoDB on one machine if you're not in hurry. - Don't be too kind on the big websites. They can afford to keep all their data in hot pages, and as a one man you will never exhaust them.
- hitpointdrew 7y ago> - Call their API through ipv6, because they may not yet have a proper, ipv6 subnet-based rate limiter. Nice tip!! > -- From a purely practical perspective, start with a baremetal transaction-isolation-less database like Cassandra/ScyllaDB. Don't rely on googling "postgres vs mongodb" or "sql vs nosql", those articles will all end in "YMMV". What you really need is massive IOPS, and a multi-node ring-based index with ScyllaDB will achieve that easily. Or just use MongoDB on one machine if you're not in hurry. Somewhat ironically Elasticsearch would probably work really well for this too (just make sure your elasticsearch isn't open to the world on the internet!).
- kaivi 7y ago>Somewhat ironically Elasticsearch would probably work really well for this too (just make sure your elasticsearch isn't open to the world on the internet!). Sure it will work, but I personally don't like Elasticsearch for anything high-intensity because of its HTTP REST API and the overhead it carries. Take a look at Cassandra's [1] "CQL binary protocol", it simple and always on point. [1] https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v4.spec https://github.com/apache/cassandra/blob/trunk/doc/native_pr...
- MrOxiMoron 7y agoonce worked on a project that tried to do just that, but at the time the LinkedIn api was already limited to seeing the authenticated users connections connections, which was too limited for what we wanted to do, can only imagine it got worse. It's also the reason recruiters really want to connect to you on LinkedIn because even if you are not interested, your connections might be.
- scraping_legal 7y agoThe US courts decided that scraping is legal, even if against EULA: > In a long-awaited decision in hiQ Labs, Inc. v. LinkedIn Corp., the Ninth Circuit Court of Appeals ruled that automated scraping of publicly accessible data likely does not violate the Computer Fraud and Abuse Act (CFAA). This is an important clarification of the CFAA’s scope, which should provide some relief to the wide variety of researchers, journalists, and companies who have had reason to fear cease and desist letters threatening liability simply for accessing publicly available information in a way that publishers object to. It’s a major win for research and innovation, which will hopefully pave the way for courts and Congress to further curb abuse of the CFAA. https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...
- shkkmo 7y agoThat is a blatant misrepresentation of that decision. That decision was upholding a lower court's preliminary injunction that prevents LinkedIn from blocking hiQ while the main case between the two is litigated. It is not a final decision and it doesn't purport to say that scraping is legal (it even points out other laws besides the CFAA that might be used to prohibit scraping.)