45 ms·

A raw dump of companies from all over the world by LinkedIn handle

197 points by mfrye0 3y ago

mfrye0 3y ago
Hey HN, we're thrilled to announce our latest project - the World's Largest Open Source Company Dataset. Our team has been working hard on this product for the past few months, and we're excited to finally share it with you all. We started off years ago trying to build a B2B app, but getting basic company data at scale was a huge barrier for us. This 15M+ record dataset attempts to solve that and has all the key company fields like name, industry, size, location, LinkedIn handle, etc. We aim to update it quarterly to ensure that you always have the most up-to-date information. Disclaimer: Okay, we have to admit, we didn't exactly comb through every dataset out there to verify that ours is the world's largest, but we did our research, and we're pretty sure it might be. Whether or not that's true, we believe this dataset is a robust and invaluable resource for anyone interested in company data.
- rickette 3y ago
  Out of curiosity: Would you be willing to share how you acquired this data? Website scraping or other means?
  - Atlas22 3y ago
    Data is very likely to be from LinkedIn if you look at the field descriptions and stats. The only field that is 100% available is the one based on the LinkedIn URL. I would guess scraping unless LinkedIn provides an API for this data that I can't find.
    - jamesgill 3y ago
      They (Microsoft) have APIs for app/website integration, but that's all I know about. https://developer.linkedin.com/ https://developer.linkedin.com/
  - yolo3000 3y ago
    Most likely from scraping (crunchbase, yahoo,etc), unless they bought it from somewhere. In most countries you can get it from the chamber of commerce. Dun and Bradstreet and other similar companies. Some of these data aggregators will have partnership with other companies, and you can also (illegally) scrape it from there.
- jsty 3y ago
  What's the license the dataset is released under? I poked around the documentation a bit but couldn't find it - apologies if it's in there!
  - conzept 3y ago
    The data is not licensed as opensource (only non-commercial use): "The Service and its entire contents, features and functionality (including but not limited to all information, software, text, displays, images, video and audio, and the design, selection and arrangement thereof), are owned by the Company, its licensors or other providers of such material and are protected by United States and international copyright, trademark, patent, trade secret and other intellectual property or proprietary rights laws. You are permitted to use the Service for Your personal, non-commercial use, or legitimate business purposes related to Your role as a customer of BigPicture.io." - https://bigpicture.io/terms https://bigpicture.io/terms
- mfrye0 3y ago
  So this blew up today. Reviewing all the comments now. 1. Yes, this is scraped from public sources. 2. Yes, this is free to use / is open source in the broadest sense. Apologies for the confusion on the lack of a license and no mention about this in our TOS. We probably should update our TOS to be clearer here. 3. This is a raw dump of companies from all over the world by LinkedIn handle. The handles are deduped, but the website is not.
  - justinclift 3y ago
    > Yes, this is free to use ... Including for commercial purposes?
    - mfrye0 3y ago
      Yes.
      - tyingq 3y ago
        Seems at odds with your tos. https://bigpicture.io/terms https://bigpicture.io/terms
data_maan 3y ago
I looked at the attributes they say the dataset has. Not too many (e.g. number of people, location). The really interesting ones, like who is doing business with whom, are missing.
- mikecoles 3y ago
  I'd like to see parent company listed. How many brands fall under TTI or Unilever? For example, a hierarchy of tool brands: https://www.protoolreviews.com/power-tool-manufacturers-who-owns-them/ https://www.protoolreviews.com/power-tool-manufacturers-who-...
  - NoboruWataya 3y ago
    https://search.gleif.org/#/record/549300MKFYEKVRWML317 https://search.gleif.org/#/record/549300MKFYEKVRWML317 (Not brands, but shows you subsidiaries.)
- cdkmoose 3y ago
  From the deeper dataset documentation, it looks like this free dataset is a field subset of their paid product.
photochemsyn 3y ago
"With over 15 million global companies included..." What distinguishes a global from a non-global company? Also, how many of these are anonymous Delaware/Nevada/South Dakota/etc-based shell companies, or are those excluded from the dataset somehow?
- Veen 3y ago
  I suspect it means "15 million companies from around the world".
- chirau 3y ago
  I think they meant 'from across the globe' as in not necessarily from the US only or other specific geographies.
1024core 3y ago
What would be really interesting is to turn this into a graph based on, say, past experience of CEOs/big dealings with each other/etc.
- tomalaci 3y ago
  The moment those graph connections will show negative view of some of those CEOs they will start screaming about privacy issues and/or sue the graph creator. With scraping LinkedIn alone it should already be possible to create such graphs. Not sure why I haven't seen any. Either scraping LinkedIn isn't as straightforward or such grap-creating attempts have been shut down due to potential for defaming someone.
  - 1024core 3y ago
    I would say that scraping LinkedIn is not easy.
visarga 3y ago
I searched for a dataset like this for a long time, trying to use it to augment named entity recognition tasks for documents. But now that GPT is on the market, this works out of the box. It's still useful as reference for validation.
andylynch 3y ago
How do you plan to identify or differentiate between legal entities? Eg a big company like your Uber example will often numerous subsidiaries, in many jurisdictions. Do you plan on including well-known identifiers like LEIs in your model?
wg0 3y ago
What I am curious to know about is - what company buys from whom and the whole dependency graph to visualise how complex our modern economy is. But not sure that kind of information is in there.
- yolo3000 3y ago
  That sort of data is normally not public
  - sam_lowry_ 3y ago
    Individual countries know this, and even share rather successfully across unions. EU exchanges VAT information which is exactly about who buys from whom... down to transactions of 120€ and more if I am not mistaken. Commercial databases are more limited, but even those can trace ownership of companies to ultimate beneficial owners. Orbis from Bureau Van Dijk is probably the biggest of such databases.
- byyll 3y ago
  Closest I can think of: https://www.importyeti.com https://www.importyeti.com
tomalaci 3y ago
How hard is it to scrape LinkedIn for all its public profile data? Do you need special developer access? Do you need to sign some contract with MSFT for anything nontrivial?
gorbachev 3y ago
Is LinkedIn scraped data open source?
nologic01 3y ago
Good luck with your launch! This reminded me of a similar project, the opencorporates database (https://opencorporates.com/ https://opencorporates.com/), though the target use cases seem different.
givemeethekeys 3y ago
Are the entries deduped? If one company owns another, is that represented as well?
simonw 3y ago
It's a 2.64GB CSV file with the following columns: handle type name website founded industry specialties size city state country_code 15,263,246 rows. I think the main listing for Google is this one (as an example): 10361050:company/google,Public Company,Google,goo.gle,,Software Development,"search, ads, mobile, android, online video, apps, machine learning, virtual reality, cloud, hardware, artificial intelligence, youtube, and software","10,001+",Mountain View,California,US
- mfrye0 3y ago
  How did you compute this? I just did another check to verify (wc -l) and it's coming to 15,980,531.
  - rahimnathwani 3y ago
    wc -l will count all newlines, even those that are escaped. Perhaps some company descriptions have newlines?
  - simonw 3y ago
    I used wc -l at first, but I've just imported into SQLite and the count(*) is 15,263,246 - updated my previous comment (which had said 15,263,251). I downloaded the CSV and ran: sqlite-utils insert companies.db company companies-dataset-2023-02-ckgENv.csv --csv sqlite-utils enable-fts companies.db company name specialties sqlite-utils analyze-tables --save companies.db This lets me run searches against the name and specialties columns, and gives me those aggregate stats too.
    - mfrye0 3y ago
      Ok. I'm not sure how this happened, but I think the dataset was somehow mislabeled. It appears that this dataset is the Q1 version, not the latest Q2. Can you please try re-downloading it? We're probably going to have to make an public announcement about this...
      - 3y ago
babblingfish 3y ago
It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all companies. If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service. There are many companies that sell data scraped from LinkedIn as a product. LinkedIn won a court case against hiQ Labs for scraping member data and other things[1]. I am not trying to compare this court case to the OP's website, just something worth mentioning. In any case, web scraping is a sort of gray area of the law. In my opinion, this data set does not contain member data and is not being monetized so it feels kosher to me. (Opinions expressed are solely my own and do not express the views or opinions of my employer.) [1] https://www2.staffingindustry.com/Editorial/IT-Staffing-Report/Jan.-5-2023/LinkedIn-ends-legal-battle-in-data-scraping-case https://www2.staffingindustry.com/Editorial/IT-Staffing-Repo...
- simonw 3y ago
  > A company has to pay to have this page, so this certainly does not include all companies. I found a listing in there for my old startup derived from the LinkedIn page, and we never paid for that LinkedIn listing.
  - babblingfish 3y ago
    Yeah apparently it's free, my bad, I remembered it incorrectly
- prepend 3y ago
  > In any case, web scraping is a sort of gray area of the law. I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.
Murrawhip 3y ago
On your home page you list Microsoft as being one of your clients. I'm pretty impressed that you managed to sell them what appears to be (mostly) their own data.
- mpeg 3y ago
  We went through this in the ad world before GDPR existed. Companies were selling datasets for advertising and, I remember one of my clients (one of the largest ad agencies in the world) telling me they had over 100 sources of data they augmented their publisher data with but they tested and saw there was only about 20% uniqueness on average between sources. They were buying the same data again and again, recycled and repackaged.
deleted 3y ago
[deleted]
ricardo81 3y ago
'open source' scrape crunchbase scrape companies house scrape wherever else scrape linkedin frontier company... or maybe not.
speedgoose 3y ago
The blog picture looks to be generated with dalle2. The quality was mind blowing less than a year ago, while it is now a mess of artefacts compared to dalle2.5 (Bing), adobe firefly, stable diffusion, and of course MidJourney.
r3trohack3r 3y ago
This is awesome and in the ballpark of something I'm working on right now. I currently have a list of developer handles, their associated aliases, and their associated email addresses - trying to map that set to employment history. Do folks know of any good data sets for this?
paxys 3y ago
You use "open source" multiple times in the post, HN title, HN comments, but: 1. The source code for the project isn't shared anywhere. 2. The data isn't shared under any standard open source license. 3. The terms of your site explicitly prohibit commercial use of this data. So what exactly makes this "open source in the broadest sense"?
- nxqs 3y ago
  "It's 'open source'"... https://youtu.be/dTRKCXC0JFg?t=6 https://youtu.be/dTRKCXC0JFg?t=6
- mfrye0 3y ago
  Fair points. I agree the wording could be better. No, the source code is not available. This dataset is a subset of the raw data our system collects. Our final product made available via the API does a variety of processing steps on the raw data (dedupes, joins, ML predictions, etc). The final, processed data is the piece that is proprietary / subject to the terms. We will update the site terms to reference this dataset, as we aim to continue releasing an updated version each quarter. I'll have to double check with the lawyers, but it will most likely be MIT licensed.
  - slabity 3y ago
    What exactly does "open-source" mean to you? Because it sounds like there's absolutely nothing open about this other than a small scraping of LinkedIn data (which you should probably ask your lawyers if you're even allowed to license out). The wording isn't just misleading, it's a complete lie. EDIT: Nevermind, the title has been updated to accurately reflect this being a small datadump.
- time_to_smile 3y ago
  It's open source in the sense of OSINT [0]. Clearly confusing on a site like Hacker News, but this has been standard usage of the term for that community for a long time now. 0. https://en.wikipedia.org/wiki/Open-source_intelligence https://en.wikipedia.org/wiki/Open-source_intelligence
deleted 3y ago
[deleted]
vrglvrglvrgl 3y ago
[dead]
deleted 3y ago
[deleted]
pimlottc 3y ago
I'm confused, most of what's in this dataset has nothing to do with RedHat.
Wronnay 3y ago
I always get "Oops! We ran into an error. Contact us at support@bigpicture.io" when I try to sign-up
- mfrye0 3y ago
  Apologies. We've had a huge problem with bots, so we have a number of security measures in place. The Google Captcha component is probably flagging you as a bot. Try disabling your VPN if you're on one, or use a different IP.
  - seanhunter 3y ago
    Oh the irony.
dang 3y ago
The submitted title was "World's largest open source company dataset", but (1) "world's largest" is linkbait and the article walks it back, (2) "open source" could be worded better per https://news.ycombinator.com/item?id=35979581 https://news.ycombinator.com/item?id=35979581, and (3) the only thing left in the title after taking those out would be "company dataset", which is too generic to be a good title. I've therefore replaced the title above with what appears to be an accurate description from https://news.ycombinator.com/item?id=35978156 https://news.ycombinator.com/item?id=35978156.
- mfrye0 3y ago
  Hey dang. That's fair. To be frank, we went back and forth on this, but in the end, thought the original title was ok. The only other large, "open source" dataset we could find was 9M. So after researching, we came to the conclusion that it sounded clickbaity, but was likely accurate. And yes on "open source". We fully intend for this to be "open source" in the full meaning of the word, but it seems we were moving too fast and missed adding the formal license.
  - deleted 3y ago
    [deleted]
companydataguy 3y ago
This is Duedil + Company Check + Open Corporates. Duedil and CC were (mostly) powered by Creditsafe data which is much better in Europe at least than D&B Open Corporates sold their data to Creditsafe for low 5 figures. Interesting point re DnB in Eu it’s mostly a license if the brand name and owns little of the data or the business.
deleted 3y ago
[deleted]
decide1000 3y ago
Not sure why you need an account. Download it here: https://bigpicture-datasets-public.s3.us-west-2.amazonaws.com/companies-dataset-2023-02-ckgENv.csv.gz https://bigpicture-datasets-public.s3.us-west-2.amazonaws.co...
- ttul 3y ago
  Oh boy, DMCA request incoming...
  - maxlin 3y ago
    That is literally their own site. I don't think they would DMCA themselves Thanks for the link. The work email requirement for just the download is just bullshit (yes I did read their piece, with that kind of logic they could also require credit card info etc while still calling it free like many scammy sites do)
- medvezhenok 3y ago
  Presumably as lead generation for their main upsell, which is the enriched version of the same data set :)
  - opportune 3y ago
    This whole thing is amateur to the core, it’s an improperly licensed (are you open source or not? What license? You don’t get to say “probably eventually MIT” lol) freemium scrape of linkedin masquerading as a product
- sundarurfriend 3y ago
  There's a comment below saying it's a 2.64GB CSV file, while gzip shows the uncompressed size of this one to be 1.97GB.
- maxlin 3y ago
  New link: https://bigpicture-datasets-public.s3.us-west-2.amazonaws.com/companies-dataset-2023-02-DzHz0U.csv.gz https://bigpicture-datasets-public.s3.us-west-2.amazonaws.co...
tuukkah 3y ago
A simple Wikidata query can return same type of information in case you prefer open data: https://w.wiki/6ify https://w.wiki/6ify
whoomp12342 3y ago
sounds fun until I have to make an account. then noep. too lazy.
borkborkimacat 3y ago
semi-direct link as there's some tomfoolery going on with this: https://wetransfer.com/downloads/b937345cd81d96654cb2d2bb43d4d97c20230518012146/45406d9077ab083a84bca5909fc425a720230518012156/cde513?trk=TRN_TDL_01&utm_campaign=TRN_TDL_01&utm_medium=email&utm_source=sendgrid https://wetransfer.com/downloads/b937345cd81d96654cb2d2bb43d...
brentis 3y ago
Can we rename to Glengarry leads?
- seanhunter 3y ago
  You don’t get the Glengarry leads. Because to give them to you … would be throwing them away.
Rastonbury 3y ago
15 million is small, for example there are vendors who have indexed several hundred million, not speaking from data quality but size wise

197 points by mfrye0 3y ago

Hey HN, we're thrilled to announce our latest project - the World's Largest Open Source Company Dataset. Our team has been working hard on this product for the past few months, and we're excited to finally share it with you all. We started off years ago trying to build a B2B app, but getting basic company data at scale was a huge barrier for us. This 15M+ record dataset attempts to solve that and has all the key company fields like name, industry, size, location, LinkedIn handle, etc. We aim to update it quarterly to ensure that you always have the most up-to-date information. Disclaimer: Okay, we have to admit, we didn't exactly comb through every dataset out there to verify that ours is the world's largest, but we did our research, and we're pretty sure it might be. Whether or not that's true, we believe this dataset is a robust and invaluable resource for anyone interested in company data.

rickette 3y ago
Out of curiosity: Would you be willing to share how you acquired this data? Website scraping or other means?
- Atlas22 3y ago
  Data is very likely to be from LinkedIn if you look at the field descriptions and stats. The only field that is 100% available is the one based on the LinkedIn URL. I would guess scraping unless LinkedIn provides an API for this data that I can't find.
  - jamesgill 3y ago
    They (Microsoft) have APIs for app/website integration, but that's all I know about. https://developer.linkedin.com/ https://developer.linkedin.com/
- yolo3000 3y ago
  Most likely from scraping (crunchbase, yahoo,etc), unless they bought it from somewhere. In most countries you can get it from the chamber of commerce. Dun and Bradstreet and other similar companies. Some of these data aggregators will have partnership with other companies, and you can also (illegally) scrape it from there.
jsty 3y ago
What's the license the dataset is released under? I poked around the documentation a bit but couldn't find it - apologies if it's in there!
- conzept 3y ago
  The data is not licensed as opensource (only non-commercial use): "The Service and its entire contents, features and functionality (including but not limited to all information, software, text, displays, images, video and audio, and the design, selection and arrangement thereof), are owned by the Company, its licensors or other providers of such material and are protected by United States and international copyright, trademark, patent, trade secret and other intellectual property or proprietary rights laws. You are permitted to use the Service for Your personal, non-commercial use, or legitimate business purposes related to Your role as a customer of BigPicture.io." - https://bigpicture.io/terms https://bigpicture.io/terms
mfrye0 3y ago
So this blew up today. Reviewing all the comments now. 1. Yes, this is scraped from public sources. 2. Yes, this is free to use / is open source in the broadest sense. Apologies for the confusion on the lack of a license and no mention about this in our TOS. We probably should update our TOS to be clearer here. 3. This is a raw dump of companies from all over the world by LinkedIn handle. The handles are deduped, but the website is not.
- justinclift 3y ago
  > Yes, this is free to use ... Including for commercial purposes?
  - mfrye0 3y ago
    Yes.
    - tyingq 3y ago
      Seems at odds with your tos. https://bigpicture.io/terms https://bigpicture.io/terms

data_maan 3y ago

I looked at the attributes they say the dataset has. Not too many (e.g. number of people, location). The really interesting ones, like who is doing business with whom, are missing.

mikecoles 3y ago
I'd like to see parent company listed. How many brands fall under TTI or Unilever? For example, a hierarchy of tool brands: https://www.protoolreviews.com/power-tool-manufacturers-who-owns-them/ https://www.protoolreviews.com/power-tool-manufacturers-who-...
- NoboruWataya 3y ago
  https://search.gleif.org/#/record/549300MKFYEKVRWML317 https://search.gleif.org/#/record/549300MKFYEKVRWML317 (Not brands, but shows you subsidiaries.)
cdkmoose 3y ago
From the deeper dataset documentation, it looks like this free dataset is a field subset of their paid product.

photochemsyn 3y ago

"With over 15 million global companies included..." What distinguishes a global from a non-global company? Also, how many of these are anonymous Delaware/Nevada/South Dakota/etc-based shell companies, or are those excluded from the dataset somehow?

Veen 3y ago
I suspect it means "15 million companies from around the world".
chirau 3y ago
I think they meant 'from across the globe' as in not necessarily from the US only or other specific geographies.

1024core 3y ago

What would be really interesting is to turn this into a graph based on, say, past experience of CEOs/big dealings with each other/etc.

tomalaci 3y ago
The moment those graph connections will show negative view of some of those CEOs they will start screaming about privacy issues and/or sue the graph creator. With scraping LinkedIn alone it should already be possible to create such graphs. Not sure why I haven't seen any. Either scraping LinkedIn isn't as straightforward or such grap-creating attempts have been shut down due to potential for defaming someone.
- 1024core 3y ago
  I would say that scraping LinkedIn is not easy.

visarga 3y ago

I searched for a dataset like this for a long time, trying to use it to augment named entity recognition tasks for documents. But now that GPT is on the market, this works out of the box. It's still useful as reference for validation.

andylynch 3y ago

How do you plan to identify or differentiate between legal entities? Eg a big company like your Uber example will often numerous subsidiaries, in many jurisdictions. Do you plan on including well-known identifiers like LEIs in your model?

wg0 3y ago

What I am curious to know about is - what company buys from whom and the whole dependency graph to visualise how complex our modern economy is. But not sure that kind of information is in there.

yolo3000 3y ago
That sort of data is normally not public
- sam_lowry_ 3y ago
  Individual countries know this, and even share rather successfully across unions. EU exchanges VAT information which is exactly about who buys from whom... down to transactions of 120€ and more if I am not mistaken. Commercial databases are more limited, but even those can trace ownership of companies to ultimate beneficial owners. Orbis from Bureau Van Dijk is probably the biggest of such databases.
byyll 3y ago
Closest I can think of: https://www.importyeti.com https://www.importyeti.com

tomalaci 3y ago

How hard is it to scrape LinkedIn for all its public profile data? Do you need special developer access? Do you need to sign some contract with MSFT for anything nontrivial?

gorbachev 3y ago

Is LinkedIn scraped data open source?

nologic01 3y ago

Good luck with your launch! This reminded me of a similar project, the opencorporates database (https://opencorporates.com/ https://opencorporates.com/), though the target use cases seem different.

givemeethekeys 3y ago

Are the entries deduped? If one company owns another, is that represented as well?

simonw 3y ago

It's a 2.64GB CSV file with the following columns: handle type name website founded industry specialties size city state country_code 15,263,246 rows. I think the main listing for Google is this one (as an example): 10361050:company/google,Public Company,Google,goo.gle,,Software Development,"search, ads, mobile, android, online video, apps, machine learning, virtual reality, cloud, hardware, artificial intelligence, youtube, and software","10,001+",Mountain View,California,US

mfrye0 3y ago
How did you compute this? I just did another check to verify (wc -l) and it's coming to 15,980,531.
- rahimnathwani 3y ago
  wc -l will count all newlines, even those that are escaped. Perhaps some company descriptions have newlines?
- simonw 3y ago
  I used wc -l at first, but I've just imported into SQLite and the count(*) is 15,263,246 - updated my previous comment (which had said 15,263,251). I downloaded the CSV and ran: sqlite-utils insert companies.db company companies-dataset-2023-02-ckgENv.csv --csv sqlite-utils enable-fts companies.db company name specialties sqlite-utils analyze-tables --save companies.db This lets me run searches against the name and specialties columns, and gives me those aggregate stats too.
  - mfrye0 3y ago
    Ok. I'm not sure how this happened, but I think the dataset was somehow mislabeled. It appears that this dataset is the Q1 version, not the latest Q2. Can you please try re-downloading it? We're probably going to have to make an public announcement about this...
    - 3y ago

babblingfish 3y ago

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all companies. If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service. There are many companies that sell data scraped from LinkedIn as a product. LinkedIn won a court case against hiQ Labs for scraping member data and other things[1]. I am not trying to compare this court case to the OP's website, just something worth mentioning. In any case, web scraping is a sort of gray area of the law. In my opinion, this data set does not contain member data and is not being monetized so it feels kosher to me. (Opinions expressed are solely my own and do not express the views or opinions of my employer.) [1] https://www2.staffingindustry.com/Editorial/IT-Staffing-Report/Jan.-5-2023/LinkedIn-ends-legal-battle-in-data-scraping-case https://www2.staffingindustry.com/Editorial/IT-Staffing-Repo...

simonw 3y ago
> A company has to pay to have this page, so this certainly does not include all companies. I found a listing in there for my old startup derived from the LinkedIn page, and we never paid for that LinkedIn listing.
- babblingfish 3y ago
  Yeah apparently it's free, my bad, I remembered it incorrectly
prepend 3y ago
> In any case, web scraping is a sort of gray area of the law. I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.

Murrawhip 3y ago

On your home page you list Microsoft as being one of your clients. I'm pretty impressed that you managed to sell them what appears to be (mostly) their own data.

mpeg 3y ago
We went through this in the ad world before GDPR existed. Companies were selling datasets for advertising and, I remember one of my clients (one of the largest ad agencies in the world) telling me they had over 100 sources of data they augmented their publisher data with but they tested and saw there was only about 20% uniqueness on average between sources. They were buying the same data again and again, recycled and repackaged.

deleted 3y ago

[deleted]

ricardo81 3y ago

'open source' scrape crunchbase scrape companies house scrape wherever else scrape linkedin frontier company... or maybe not.

speedgoose 3y ago

The blog picture looks to be generated with dalle2. The quality was mind blowing less than a year ago, while it is now a mess of artefacts compared to dalle2.5 (Bing), adobe firefly, stable diffusion, and of course MidJourney.

r3trohack3r 3y ago

This is awesome and in the ballpark of something I'm working on right now. I currently have a list of developer handles, their associated aliases, and their associated email addresses - trying to map that set to employment history. Do folks know of any good data sets for this?

paxys 3y ago

You use "open source" multiple times in the post, HN title, HN comments, but: 1. The source code for the project isn't shared anywhere. 2. The data isn't shared under any standard open source license. 3. The terms of your site explicitly prohibit commercial use of this data. So what exactly makes this "open source in the broadest sense"?

nxqs 3y ago
"It's 'open source'"... https://youtu.be/dTRKCXC0JFg?t=6 https://youtu.be/dTRKCXC0JFg?t=6
mfrye0 3y ago
Fair points. I agree the wording could be better. No, the source code is not available. This dataset is a subset of the raw data our system collects. Our final product made available via the API does a variety of processing steps on the raw data (dedupes, joins, ML predictions, etc). The final, processed data is the piece that is proprietary / subject to the terms. We will update the site terms to reference this dataset, as we aim to continue releasing an updated version each quarter. I'll have to double check with the lawyers, but it will most likely be MIT licensed.
- slabity 3y ago
  What exactly does "open-source" mean to you? Because it sounds like there's absolutely nothing open about this other than a small scraping of LinkedIn data (which you should probably ask your lawyers if you're even allowed to license out). The wording isn't just misleading, it's a complete lie. EDIT: Nevermind, the title has been updated to accurately reflect this being a small datadump.
time_to_smile 3y ago
It's open source in the sense of OSINT [0]. Clearly confusing on a site like Hacker News, but this has been standard usage of the term for that community for a long time now. 0. https://en.wikipedia.org/wiki/Open-source_intelligence https://en.wikipedia.org/wiki/Open-source_intelligence

deleted 3y ago

[deleted]

vrglvrglvrgl 3y ago

[dead]

deleted 3y ago

[deleted]

pimlottc 3y ago

I'm confused, most of what's in this dataset has nothing to do with RedHat.

Wronnay 3y ago

I always get "Oops! We ran into an error. Contact us at support@bigpicture.io" when I try to sign-up

mfrye0 3y ago
Apologies. We've had a huge problem with bots, so we have a number of security measures in place. The Google Captcha component is probably flagging you as a bot. Try disabling your VPN if you're on one, or use a different IP.
- seanhunter 3y ago
  Oh the irony.

dang 3y ago

The submitted title was "World's largest open source company dataset", but (1) "world's largest" is linkbait and the article walks it back, (2) "open source" could be worded better per https://news.ycombinator.com/item?id=35979581 https://news.ycombinator.com/item?id=35979581, and (3) the only thing left in the title after taking those out would be "company dataset", which is too generic to be a good title. I've therefore replaced the title above with what appears to be an accurate description from https://news.ycombinator.com/item?id=35978156 https://news.ycombinator.com/item?id=35978156.

mfrye0 3y ago
Hey dang. That's fair. To be frank, we went back and forth on this, but in the end, thought the original title was ok. The only other large, "open source" dataset we could find was 9M. So after researching, we came to the conclusion that it sounded clickbaity, but was likely accurate. And yes on "open source". We fully intend for this to be "open source" in the full meaning of the word, but it seems we were moving too fast and missed adding the formal license.
- deleted 3y ago
  [deleted]

companydataguy 3y ago

This is Duedil + Company Check + Open Corporates. Duedil and CC were (mostly) powered by Creditsafe data which is much better in Europe at least than D&B Open Corporates sold their data to Creditsafe for low 5 figures. Interesting point re DnB in Eu it’s mostly a license if the brand name and owns little of the data or the business.

deleted 3y ago

[deleted]

decide1000 3y ago

Not sure why you need an account. Download it here: https://bigpicture-datasets-public.s3.us-west-2.amazonaws.com/companies-dataset-2023-02-ckgENv.csv.gz https://bigpicture-datasets-public.s3.us-west-2.amazonaws.co...

ttul 3y ago
Oh boy, DMCA request incoming...
- maxlin 3y ago
  That is literally their own site. I don't think they would DMCA themselves Thanks for the link. The work email requirement for just the download is just bullshit (yes I did read their piece, with that kind of logic they could also require credit card info etc while still calling it free like many scammy sites do)
medvezhenok 3y ago
Presumably as lead generation for their main upsell, which is the enriched version of the same data set :)
- opportune 3y ago
  This whole thing is amateur to the core, it’s an improperly licensed (are you open source or not? What license? You don’t get to say “probably eventually MIT” lol) freemium scrape of linkedin masquerading as a product
sundarurfriend 3y ago
There's a comment below saying it's a 2.64GB CSV file, while gzip shows the uncompressed size of this one to be 1.97GB.
maxlin 3y ago
New link: https://bigpicture-datasets-public.s3.us-west-2.amazonaws.com/companies-dataset-2023-02-DzHz0U.csv.gz https://bigpicture-datasets-public.s3.us-west-2.amazonaws.co...

tuukkah 3y ago

A simple Wikidata query can return same type of information in case you prefer open data: https://w.wiki/6ify https://w.wiki/6ify

whoomp12342 3y ago

sounds fun until I have to make an account. then noep. too lazy.

borkborkimacat 3y ago

semi-direct link as there's some tomfoolery going on with this: https://wetransfer.com/downloads/b937345cd81d96654cb2d2bb43d4d97c20230518012146/45406d9077ab083a84bca5909fc425a720230518012156/cde513?trk=TRN_TDL_01&utm_campaign=TRN_TDL_01&utm_medium=email&utm_source=sendgrid https://wetransfer.com/downloads/b937345cd81d96654cb2d2bb43d...

brentis 3y ago

Can we rename to Glengarry leads?

seanhunter 3y ago
You don’t get the Glengarry leads. Because to give them to you … would be throwing them away.

Rastonbury 3y ago

15 million is small, for example there are vendors who have indexed several hundred million, not speaking from data quality but size wise