5 ms·

Discovering Millions of Datasets on the Web

443 points by Anon84 7y ago

dzonga 7y ago
I really feel like the data side of things is under-rated. mostly, it seems like when people talk of IP, they talk about th e software and forget the data. Uber, Snapchat etc are companies mostly in the business of shuffling data around. Good or bad, that's subjective. And this data-search product is a nice welcome to those people who are trying to get something off the ground, research or just trying to understand the world and human behaviour better.
- bordercases 7y ago
  I don't think it's underrated, but I do think that there is a gap between massive data and an idea of what to do with it. And data is simple, it's parameters plus timestamps plus a lot of storage. Realtime access is harder but it's a well-specified problem. The issue is inference. No one does inference extremely well except in limited circumstances. It's one of our greatest bottlenecks as humans and our software is going to be limited by it as well insofar as our understanding of what to build is controlled by what kind of inference we want.
  - danso 7y ago
    I think data is underrated because it is actually not "simple". Especially the collection and curation of it.
- jimkri 7y ago
  I agree that data is underrated. The commercial real estate industry is a really interesting example. There are a very limited number of companies that have complete datasets of buildings in all cities throughout the country. The leader is CoStar, they have the most complete and accurate data. What I have noticed with other CRE data companies is they are focusing on leasing activity or 1 other part of the CRE process. CoStar became the leader by valuing their data over all things, if other companies want to compete they need to do the same.
- anthonypasq 7y ago
  How is data underrated? Everything is data. Data is the primary asset of most technology companies.
blacksmith_tb 7y ago
I was picturing a link to Shodan, nice to see this is about legitimate sources instead.
sixstringtheory 7y ago
This is great. I've been collecting a list of open data sets for a while now with an eye to at some point turn it into a blog post. Now maybe I don't have to... saved me some work! Some other indices of open data sets I've found: https://registry.opendata.aws https://registry.opendata.aws https://en.m.wikipedia.org/wiki/List_of_datasets_for_machine-learning_research https://en.m.wikipedia.org/wiki/List_of_datasets_for_machine... https://meta.m.wikimedia.org/wiki/Datasets https://meta.m.wikimedia.org/wiki/Datasets
- breck 7y ago
  This is a good list: https://github.com/awesomedata/awesome-public-datasets https://github.com/awesomedata/awesome-public-datasets
  - subroutine 7y ago
    Also zenodo.org is seems like it's gaining traction as a de facto data repo for scientific journals. I've had to deposit a copy of my raw data here 2x in the last 6 months for different pubs (1 neurobio, 1 genomics).
livingmargot 7y ago
Still can't find the damned 'International Corpus of Learned English', though.
- nl 7y ago
  You seem to be able to order it here: https://www.i6doc.com/en/collections/cdicle/ https://www.i6doc.com/en/collections/cdicle/
  - livingmargot 7y ago
    I have no practical way of reading a CD-ROM, why don't they make available a virtual version? Thanks anyway
    - ant6n 7y ago
      If you pay 350$ for a cd rom u should be able to find a way to digitize it, for example a copy shop or just getting a usb drive.
    - nl 7y ago
      A USB CDROM drive is ~$20.
- kristianp 7y ago
  Actually learner English. Sounds like an interesting corpus to work on.
- philshem 7y ago
  is this it? https://opendata.stackexchange.com/a/4308/1511 https://opendata.stackexchange.com/a/4308/1511 http://www.hlt.utdallas.edu/~persingq/ICLE/orgDataset.html http://www.hlt.utdallas.edu/~persingq/ICLE/orgDataset.html http://www.hlt.utdallas.edu/~persingq/ICLE/OrganizationScores.txt http://www.hlt.utdallas.edu/~persingq/ICLE/OrganizationScore...
mqus 7y ago
This is really nice! But since this is Google: how long will it take until this gets shut down?
lettergram 7y ago
For those interested, I recently wrote a blog post on how to download & parse USPTO patents for a large free corpus for NLP problems: https://austingwalters.com/parsing-uspto-patents-to-create-a-dataset/ https://austingwalters.com/parsing-uspto-patents-to-create-a... I actually have found FOIA requests[1] and downloads from government websites to be the easiest & most effective way to get robust datasets. [1] https://austingwalters.com/foia-requesting-100-universities/ https://austingwalters.com/foia-requesting-100-universities/
grogenaut 7y ago
Trying to teach my wife pandas and the thing she most wants to do is compute the 10 year projected return on fortune 500s (buffetology) based on last ten year financial reports. It's really hard to find a good data source though as it's either in PDF or Google has been optimized to rent seeking data repackagers where it's hard to see if they have the data without jumping through hoops. Would love a source for that.
- withdavidli 7y ago
  I watch Aswath Damodaran's videos, Professor of Valuation at NYU. He recently had a video on the data he uses [0]. Might be worth it to email him to see if the data sources he buys matches your needs. [0] https://youtu.be/M9pFTApeo_8 https://youtu.be/M9pFTApeo_8 Additional link to his data site: http://people.stern.nyu.edu/adamodar/New_Home_Page/data.html http://people.stern.nyu.edu/adamodar/New_Home_Page/data.html
- texasbigdata 7y ago
  Capital IQ
- deleted 7y ago
  [deleted]
- marc__1 7y ago
  Look at the SEC/EDGAR page where you can find the data in xml and json formats https://www.sec.gov/edgar/searchedgar/accessing-edgar-data.htm https://www.sec.gov/edgar/searchedgar/accessing-edgar-data.h...
- londons_explore 7y ago
  Grab the numbers from a few of the pdf files, then do Google searches for those exact numbers and see if you can find one of those "auto-generated news sites" that shows the same numbers and scrape that?
- hbcondo714 7y ago
  If you use Google's Dataset Search for SEC Filings[1], you get outdated information. FTP access has been removed for years but SEC Filings are still are great example of large datasets. I built a side business at https://Last10K.com https://Last10K.com using buffettology and provide 10 years of company annual reports (10Ks). There's also an API at https://dev.Last10K.com https://dev.Last10K.com that returns financial data from these filings in JSON or XML. [1]https://datasetsearch.research.google.com/search?query=EDGAR%20Database%20of%20SEC%20Filings&docid=3%2Bc8oAy8U5sVy2HZAAAAAA%3D%3D https://datasetsearch.research.google.com/search?query=EDGAR...
sandinmyjoints 7y ago
Please try searching for datasets on this site on mobile. It needs some work.
philshem 7y ago
A couple good resources for finding datasets: + For individual requests, come over to https://opendata.stackexchange.com/ https://opendata.stackexchange.com/ and ask! + Wikidata has loads of structured data, but using SPARQL is often a barrier. But you can request help: https://www.wikidata.org/wiki/Wikidata:Request_a_query https://www.wikidata.org/wiki/Wikidata:Request_a_query
fudged71 7y ago
All this (and comments) have taught me is the data set I'm looking for doesn't exist in the public domain. Time to make it myself.
bg117 7y ago
Are there any resources where models are available?
davedx 7y ago
I gave this a try for a few queries, but the results are very varied. Often you get the landing page for a study with its PDF behind a research/journal paywall (even with the "Free" filter applied, so not sure what "free" means to Google). Sometimes the "dataset" is some kind of visualization without any obvious way to get the raw data. Only a couple of results had a JSON or CSV to download. Overall a bit underwhelming.
igravious 7y ago
Found a Vocabulary of Philosophy using it, very skookum! https://www.loterre.fr/skosmos/73G/en/ https://www.loterre.fr/skosmos/73G/en/ Unfortunately, most every result for the word `philosophy' is borderline garbage imho. Keyword indexing of datasets may need improving?
unitykid9008 7y ago
If anyone is looking for cleaned and linked finance datasets, and works at a university. You should double check if you get access to Wharton Research Data Services https://wrds-www.wharton.upenn.edu/ https://wrds-www.wharton.upenn.edu/, it could save you alot of time.

443 points by Anon84 7y ago

I really feel like the data side of things is under-rated. mostly, it seems like when people talk of IP, they talk about th e software and forget the data. Uber, Snapchat etc are companies mostly in the business of shuffling data around. Good or bad, that's subjective. And this data-search product is a nice welcome to those people who are trying to get something off the ground, research or just trying to understand the world and human behaviour better.

bordercases 7y ago
I don't think it's underrated, but I do think that there is a gap between massive data and an idea of what to do with it. And data is simple, it's parameters plus timestamps plus a lot of storage. Realtime access is harder but it's a well-specified problem. The issue is inference. No one does inference extremely well except in limited circumstances. It's one of our greatest bottlenecks as humans and our software is going to be limited by it as well insofar as our understanding of what to build is controlled by what kind of inference we want.
- danso 7y ago
  I think data is underrated because it is actually not "simple". Especially the collection and curation of it.
jimkri 7y ago
I agree that data is underrated. The commercial real estate industry is a really interesting example. There are a very limited number of companies that have complete datasets of buildings in all cities throughout the country. The leader is CoStar, they have the most complete and accurate data. What I have noticed with other CRE data companies is they are focusing on leasing activity or 1 other part of the CRE process. CoStar became the leader by valuing their data over all things, if other companies want to compete they need to do the same.
anthonypasq 7y ago
How is data underrated? Everything is data. Data is the primary asset of most technology companies.

blacksmith_tb 7y ago

I was picturing a link to Shodan, nice to see this is about legitimate sources instead.

sixstringtheory 7y ago

This is great. I've been collecting a list of open data sets for a while now with an eye to at some point turn it into a blog post. Now maybe I don't have to... saved me some work! Some other indices of open data sets I've found: https://registry.opendata.aws https://registry.opendata.aws https://en.m.wikipedia.org/wiki/List_of_datasets_for_machine-learning_research https://en.m.wikipedia.org/wiki/List_of_datasets_for_machine... https://meta.m.wikimedia.org/wiki/Datasets https://meta.m.wikimedia.org/wiki/Datasets

breck 7y ago
This is a good list: https://github.com/awesomedata/awesome-public-datasets https://github.com/awesomedata/awesome-public-datasets
- subroutine 7y ago
  Also zenodo.org is seems like it's gaining traction as a de facto data repo for scientific journals. I've had to deposit a copy of my raw data here 2x in the last 6 months for different pubs (1 neurobio, 1 genomics).

livingmargot 7y ago

Still can't find the damned 'International Corpus of Learned English', though.

nl 7y ago
You seem to be able to order it here: https://www.i6doc.com/en/collections/cdicle/ https://www.i6doc.com/en/collections/cdicle/
- livingmargot 7y ago
  I have no practical way of reading a CD-ROM, why don't they make available a virtual version? Thanks anyway
  - ant6n 7y ago
    If you pay 350$ for a cd rom u should be able to find a way to digitize it, for example a copy shop or just getting a usb drive.
  - nl 7y ago
    A USB CDROM drive is ~$20.
kristianp 7y ago
Actually learner English. Sounds like an interesting corpus to work on.
philshem 7y ago
is this it? https://opendata.stackexchange.com/a/4308/1511 https://opendata.stackexchange.com/a/4308/1511 http://www.hlt.utdallas.edu/~persingq/ICLE/orgDataset.html http://www.hlt.utdallas.edu/~persingq/ICLE/orgDataset.html http://www.hlt.utdallas.edu/~persingq/ICLE/OrganizationScores.txt http://www.hlt.utdallas.edu/~persingq/ICLE/OrganizationScore...

mqus 7y ago

This is really nice! But since this is Google: how long will it take until this gets shut down?

lettergram 7y ago

For those interested, I recently wrote a blog post on how to download & parse USPTO patents for a large free corpus for NLP problems: https://austingwalters.com/parsing-uspto-patents-to-create-a-dataset/ https://austingwalters.com/parsing-uspto-patents-to-create-a... I actually have found FOIA requests[1] and downloads from government websites to be the easiest & most effective way to get robust datasets. [1] https://austingwalters.com/foia-requesting-100-universities/ https://austingwalters.com/foia-requesting-100-universities/

grogenaut 7y ago

Trying to teach my wife pandas and the thing she most wants to do is compute the 10 year projected return on fortune 500s (buffetology) based on last ten year financial reports. It's really hard to find a good data source though as it's either in PDF or Google has been optimized to rent seeking data repackagers where it's hard to see if they have the data without jumping through hoops. Would love a source for that.

withdavidli 7y ago
I watch Aswath Damodaran's videos, Professor of Valuation at NYU. He recently had a video on the data he uses [0]. Might be worth it to email him to see if the data sources he buys matches your needs. [0] https://youtu.be/M9pFTApeo_8 https://youtu.be/M9pFTApeo_8 Additional link to his data site: http://people.stern.nyu.edu/adamodar/New_Home_Page/data.html http://people.stern.nyu.edu/adamodar/New_Home_Page/data.html
texasbigdata 7y ago
Capital IQ
deleted 7y ago
[deleted]
marc__1 7y ago
Look at the SEC/EDGAR page where you can find the data in xml and json formats https://www.sec.gov/edgar/searchedgar/accessing-edgar-data.htm https://www.sec.gov/edgar/searchedgar/accessing-edgar-data.h...
londons_explore 7y ago
Grab the numbers from a few of the pdf files, then do Google searches for those exact numbers and see if you can find one of those "auto-generated news sites" that shows the same numbers and scrape that?
hbcondo714 7y ago
If you use Google's Dataset Search for SEC Filings[1], you get outdated information. FTP access has been removed for years but SEC Filings are still are great example of large datasets. I built a side business at https://Last10K.com https://Last10K.com using buffettology and provide 10 years of company annual reports (10Ks). There's also an API at https://dev.Last10K.com https://dev.Last10K.com that returns financial data from these filings in JSON or XML. [1]https://datasetsearch.research.google.com/search?query=EDGAR%20Database%20of%20SEC%20Filings&docid=3%2Bc8oAy8U5sVy2HZAAAAAA%3D%3D https://datasetsearch.research.google.com/search?query=EDGAR...

sandinmyjoints 7y ago

Please try searching for datasets on this site on mobile. It needs some work.

philshem 7y ago

A couple good resources for finding datasets: + For individual requests, come over to https://opendata.stackexchange.com/ https://opendata.stackexchange.com/ and ask! + Wikidata has loads of structured data, but using SPARQL is often a barrier. But you can request help: https://www.wikidata.org/wiki/Wikidata:Request_a_query https://www.wikidata.org/wiki/Wikidata:Request_a_query

fudged71 7y ago

All this (and comments) have taught me is the data set I'm looking for doesn't exist in the public domain. Time to make it myself.

bg117 7y ago

Are there any resources where models are available?

davedx 7y ago

I gave this a try for a few queries, but the results are very varied. Often you get the landing page for a study with its PDF behind a research/journal paywall (even with the "Free" filter applied, so not sure what "free" means to Google). Sometimes the "dataset" is some kind of visualization without any obvious way to get the raw data. Only a couple of results had a JSON or CSV to download. Overall a bit underwhelming.

igravious 7y ago

Found a Vocabulary of Philosophy using it, very skookum! https://www.loterre.fr/skosmos/73G/en/ https://www.loterre.fr/skosmos/73G/en/ Unfortunately, most every result for the word `philosophy' is borderline garbage imho. Keyword indexing of datasets may need improving?

unitykid9008 7y ago

If anyone is looking for cleaned and linked finance datasets, and works at a university. You should double check if you get access to Wharton Research Data Services https://wrds-www.wharton.upenn.edu/ https://wrds-www.wharton.upenn.edu/, it could save you alot of time.