4 ms·

Hey HN, we're thrilled to announce our latest project - the World's Largest Open Source Company Dataset. Our team has been working hard on this product for the

by mfrye0 3y ago

Hey HN, we're thrilled to announce our latest project - the World's Largest Open Source Company Dataset. Our team has been working hard on this product for the past few months, and we're excited to finally share it with you all. We started off years ago trying to build a B2B app, but getting basic company data at scale was a huge barrier for us. This 15M+ record dataset attempts to solve that and has all the key company fields like name, industry, size, location, LinkedIn handle, etc. We aim to update it quarterly to ensure that you always have the most up-to-date information. Disclaimer: Okay, we have to admit, we didn't exactly comb through every dataset out there to verify that ours is the world's largest, but we did our research, and we're pretty sure it might be. Whether or not that's true, we believe this dataset is a robust and invaluable resource for anyone interested in company data.

rickette 3y ago
Out of curiosity: Would you be willing to share how you acquired this data? Website scraping or other means?
- Atlas22 3y ago
  Data is very likely to be from LinkedIn if you look at the field descriptions and stats. The only field that is 100% available is the one based on the LinkedIn URL. I would guess scraping unless LinkedIn provides an API for this data that I can't find.
  - jamesgill 3y ago
    They (Microsoft) have APIs for app/website integration, but that's all I know about. https://developer.linkedin.com/ https://developer.linkedin.com/
- yolo3000 3y ago
  Most likely from scraping (crunchbase, yahoo,etc), unless they bought it from somewhere. In most countries you can get it from the chamber of commerce. Dun and Bradstreet and other similar companies. Some of these data aggregators will have partnership with other companies, and you can also (illegally) scrape it from there.
jsty 3y ago
What's the license the dataset is released under? I poked around the documentation a bit but couldn't find it - apologies if it's in there!
- conzept 3y ago
  The data is not licensed as opensource (only non-commercial use): "The Service and its entire contents, features and functionality (including but not limited to all information, software, text, displays, images, video and audio, and the design, selection and arrangement thereof), are owned by the Company, its licensors or other providers of such material and are protected by United States and international copyright, trademark, patent, trade secret and other intellectual property or proprietary rights laws. You are permitted to use the Service for Your personal, non-commercial use, or legitimate business purposes related to Your role as a customer of BigPicture.io." - https://bigpicture.io/terms https://bigpicture.io/terms
mfrye0 3y ago
So this blew up today. Reviewing all the comments now. 1. Yes, this is scraped from public sources. 2. Yes, this is free to use / is open source in the broadest sense. Apologies for the confusion on the lack of a license and no mention about this in our TOS. We probably should update our TOS to be clearer here. 3. This is a raw dump of companies from all over the world by LinkedIn handle. The handles are deduped, but the website is not.
- justinclift 3y ago
  > Yes, this is free to use ... Including for commercial purposes?
  - mfrye0 3y ago
    Yes.
    - tyingq 3y ago
      Seems at odds with your tos. https://bigpicture.io/terms https://bigpicture.io/terms

by mfrye0 3y ago

rickette 3y ago

Out of curiosity: Would you be willing to share how you acquired this data? Website scraping or other means?

Atlas22 3y ago
Data is very likely to be from LinkedIn if you look at the field descriptions and stats. The only field that is 100% available is the one based on the LinkedIn URL. I would guess scraping unless LinkedIn provides an API for this data that I can't find.
- jamesgill 3y ago
  They (Microsoft) have APIs for app/website integration, but that's all I know about. https://developer.linkedin.com/ https://developer.linkedin.com/
yolo3000 3y ago
Most likely from scraping (crunchbase, yahoo,etc), unless they bought it from somewhere. In most countries you can get it from the chamber of commerce. Dun and Bradstreet and other similar companies. Some of these data aggregators will have partnership with other companies, and you can also (illegally) scrape it from there.

jsty 3y ago

What's the license the dataset is released under? I poked around the documentation a bit but couldn't find it - apologies if it's in there!

conzept 3y ago
The data is not licensed as opensource (only non-commercial use): "The Service and its entire contents, features and functionality (including but not limited to all information, software, text, displays, images, video and audio, and the design, selection and arrangement thereof), are owned by the Company, its licensors or other providers of such material and are protected by United States and international copyright, trademark, patent, trade secret and other intellectual property or proprietary rights laws. You are permitted to use the Service for Your personal, non-commercial use, or legitimate business purposes related to Your role as a customer of BigPicture.io." - https://bigpicture.io/terms https://bigpicture.io/terms

mfrye0 3y ago

So this blew up today. Reviewing all the comments now. 1. Yes, this is scraped from public sources. 2. Yes, this is free to use / is open source in the broadest sense. Apologies for the confusion on the lack of a license and no mention about this in our TOS. We probably should update our TOS to be clearer here. 3. This is a raw dump of companies from all over the world by LinkedIn handle. The handles are deduped, but the website is not.

justinclift 3y ago
> Yes, this is free to use ... Including for commercial purposes?
- mfrye0 3y ago
  Yes.
  - tyingq 3y ago
    Seems at odds with your tos. https://bigpicture.io/terms https://bigpicture.io/terms