19 ms·

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all companies. If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service. There are many companies that sell data scraped from LinkedIn as a product. LinkedIn won a court case against hiQ Labs for scraping member data and other things[1]. I am not trying to compare this court case to the OP's website, just something worth mentioning. In any case, web scraping is a sort of gray area of the law. In my opinion, this data set does not contain member data and is not being monetized so it feels kosher to me. (Opinions expressed are solely my own and do not express the views or opinions of my employer.) [1] https://www2.staffingindustry.com/Editorial/IT-Staffing-Report/Jan.-5-2023/LinkedIn-ends-legal-battle-in-data-scraping-case https://www2.staffingindustry.com/Editorial/IT-Staffing-Repo...

simonw 3y ago
> A company has to pay to have this page, so this certainly does not include all companies. I found a listing in there for my old startup derived from the LinkedIn page, and we never paid for that LinkedIn listing.
- babblingfish 3y ago
  Yeah apparently it's free, my bad, I remembered it incorrectly
prepend 3y ago
> In any case, web scraping is a sort of gray area of the law. I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.
- JohnFen 3y ago
  It is a gray area (in the US) in the sense that there is no clear consensus about it in the courts. There have been court rulings in both directions.
  - hallqv 3y ago
    What rulings are you referring to? Been reaffirmed that scraping LinkedIn is legal multiple times, even by Supreme Court. https://techcrunch.com/2022/04/18/web-scraping-legal-court/ https://techcrunch.com/2022/04/18/web-scraping-legal-court/
    - JohnFen 3y ago
      I believe that decision was that web scraping doesn't violate the CFAA unless there are access controls (such as the need to log in) to get to it, but they specifically said that the ruling doesn't comment about other possible claims against it. That's why I think it's still a gray area. But I could be wrong -- I haven't been following this stuff all that closely.
  - SnowHill9902 3y ago
    So it’s not gray, it’s melange.
    - hallqv 3y ago
      Multiple court reaffirms in favor of scraping, including by the SC, seems more like verdant to me..
  - prepend 3y ago
    What rulings have been against it? You don’t need consensus to allow something, you need consensus to be against something, otherwise it’s allowed. This “scraping is Grey” sounds like FUD to me. Legally, you can scrape anything publicly available as long as you damage the server. You understand that google is just a giant web scraper right? And all the other search engines spidering the web. Do you think they are risking their business being in the “grey?”
mcenedella 3y ago
LinkedIn lost the HiQ case. Your GC claiming it’s a win doesn’t change that you lost, repeatedly, in multiple courts of law: https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
- foolswisdom 3y ago
  According to that wiki page, the courts eventually did rule that HiQ violated LinkedIn's terms of service, and a settlement agreement was reached.
- henryfjordan 3y ago
  Linkedin lost on the CFAA claims, there were others that they did not lose.
- noizejoy 3y ago
  > LinkedIn lost the HiQ case. Your GC claiming it’s a win doesn’t change that you lost, repeatedly, in multiple courts of law That would appear to be false, since after appeals, there was a settlement wich would seem to make it pretty clear that hiQ lost: https://www.natlawreview.com/article/hiq-and-linkedin-reach-proposed-settlement-landmark-scraping-case https://www.natlawreview.com/article/hiq-and-linkedin-reach-...
dheera 3y ago
> In any case, web scraping is a sort of gray area of the law. I feel like it's also only a matter of time before we have decentralized web scrapers that can consume data and assemble datasets and store the results in a decentralized fashion in a way that is completely non-enforceable. LinkedIn data would be aggregated in all sorts of interesting ways, and there would be literally no name you could sue, no address you could serve papers to, it would be a hundred thousand bots in 100 countries that collectively assembled the dataset and put it on some IPFS or torrent. If data is visible with human eyes, it's visible by decentralized bots, the only missing piece is the technical and financial complexity of the current state of decentralized compute.
oldtownroad 3y ago
I have lots of LinkedIn pages for various companies (alive and dead). I’ve never had to pay: while there’s limits on page creation for free accounts (which appear to be prevent abuse) there’s no direct payment required to create a company page. I’m not sure if you’re accidentally leaking a new strategy that LinkedIn are about to launch or if you’re mistaken but given you’re claiming to work for LinkedIn, probably worth correcting your comment either way.
- babblingfish 3y ago
  Definitely not the latter. I am referring to the former point.
  - oldtownroad 3y ago
    The limit only applies if you’ve created… I can’t remember the number exactly and it’s not listed anywhere but I recall hitting a limit around 5 pages. You can see that it’s free to make a page here — the only requirement is a free account: https://www.linkedin.com/help/linkedin/answer/a543852/create-a-linkedin-page?lang=en https://www.linkedin.com/help/linkedin/answer/a543852/create...
    - babblingfish 3y ago
      TIL
B1FF_PSUVM 3y ago
> and is not being monetized It seems to have a paid version with more data: https://docs.bigpicture.io/docs/free-datasets/companies/#field-comparison-to-paid-dataset https://docs.bigpicture.io/docs/free-datasets/companies/#fie... (just the fact, I don't have an opinion)
- joshspankit 3y ago
  I’m on the same page. It also requires users to create an account in order to get the free version, and I would argue that in most cases that’s part of a monetization funnel.
- babblingfish 3y ago
  :O
SXX 3y ago
> If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service. Well. This is certainly not true. Even with Premium subscription you can hit a hard limit of personal profile views while just actively going through profiles without any automation. And Linkedin will temporary restrict and threaten to ban your account.
onetokeoverthe 3y ago
[dead]

simonw 3y ago

> A company has to pay to have this page, so this certainly does not include all companies. I found a listing in there for my old startup derived from the LinkedIn page, and we never paid for that LinkedIn listing.

babblingfish 3y ago
Yeah apparently it's free, my bad, I remembered it incorrectly

prepend 3y ago

> In any case, web scraping is a sort of gray area of the law. I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.

JohnFen 3y ago
It is a gray area (in the US) in the sense that there is no clear consensus about it in the courts. There have been court rulings in both directions.
- hallqv 3y ago
  What rulings are you referring to? Been reaffirmed that scraping LinkedIn is legal multiple times, even by Supreme Court. https://techcrunch.com/2022/04/18/web-scraping-legal-court/ https://techcrunch.com/2022/04/18/web-scraping-legal-court/
  - JohnFen 3y ago
    I believe that decision was that web scraping doesn't violate the CFAA unless there are access controls (such as the need to log in) to get to it, but they specifically said that the ruling doesn't comment about other possible claims against it. That's why I think it's still a gray area. But I could be wrong -- I haven't been following this stuff all that closely.
- SnowHill9902 3y ago
  So it’s not gray, it’s melange.
  - hallqv 3y ago
    Multiple court reaffirms in favor of scraping, including by the SC, seems more like verdant to me..
- prepend 3y ago
  What rulings have been against it? You don’t need consensus to allow something, you need consensus to be against something, otherwise it’s allowed. This “scraping is Grey” sounds like FUD to me. Legally, you can scrape anything publicly available as long as you damage the server. You understand that google is just a giant web scraper right? And all the other search engines spidering the web. Do you think they are risking their business being in the “grey?”

mcenedella 3y ago

LinkedIn lost the HiQ case. Your GC claiming it’s a win doesn’t change that you lost, repeatedly, in multiple courts of law: https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

foolswisdom 3y ago
According to that wiki page, the courts eventually did rule that HiQ violated LinkedIn's terms of service, and a settlement agreement was reached.
henryfjordan 3y ago
Linkedin lost on the CFAA claims, there were others that they did not lose.
noizejoy 3y ago
> LinkedIn lost the HiQ case. Your GC claiming it’s a win doesn’t change that you lost, repeatedly, in multiple courts of law That would appear to be false, since after appeals, there was a settlement wich would seem to make it pretty clear that hiQ lost: https://www.natlawreview.com/article/hiq-and-linkedin-reach-proposed-settlement-landmark-scraping-case https://www.natlawreview.com/article/hiq-and-linkedin-reach-...

dheera 3y ago

> In any case, web scraping is a sort of gray area of the law. I feel like it's also only a matter of time before we have decentralized web scrapers that can consume data and assemble datasets and store the results in a decentralized fashion in a way that is completely non-enforceable. LinkedIn data would be aggregated in all sorts of interesting ways, and there would be literally no name you could sue, no address you could serve papers to, it would be a hundred thousand bots in 100 countries that collectively assembled the dataset and put it on some IPFS or torrent. If data is visible with human eyes, it's visible by decentralized bots, the only missing piece is the technical and financial complexity of the current state of decentralized compute.

oldtownroad 3y ago

I have lots of LinkedIn pages for various companies (alive and dead). I’ve never had to pay: while there’s limits on page creation for free accounts (which appear to be prevent abuse) there’s no direct payment required to create a company page. I’m not sure if you’re accidentally leaking a new strategy that LinkedIn are about to launch or if you’re mistaken but given you’re claiming to work for LinkedIn, probably worth correcting your comment either way.

babblingfish 3y ago
Definitely not the latter. I am referring to the former point.
- oldtownroad 3y ago
  The limit only applies if you’ve created… I can’t remember the number exactly and it’s not listed anywhere but I recall hitting a limit around 5 pages. You can see that it’s free to make a page here — the only requirement is a free account: https://www.linkedin.com/help/linkedin/answer/a543852/create-a-linkedin-page?lang=en https://www.linkedin.com/help/linkedin/answer/a543852/create...
  - babblingfish 3y ago
    TIL

B1FF_PSUVM 3y ago

> and is not being monetized It seems to have a paid version with more data: https://docs.bigpicture.io/docs/free-datasets/companies/#field-comparison-to-paid-dataset https://docs.bigpicture.io/docs/free-datasets/companies/#fie... (just the fact, I don't have an opinion)

joshspankit 3y ago
I’m on the same page. It also requires users to create an account in order to get the free version, and I would argue that in most cases that’s part of a monetization funnel.
babblingfish 3y ago
:O

SXX 3y ago

> If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service. Well. This is certainly not true. Even with Premium subscription you can hit a hard limit of personal profile views while just actively going through profiles without any automation. And Linkedin will temporary restrict and threaten to ban your account.

onetokeoverthe 3y ago

[dead]