6 ms·
List of high-quality open datasets in public domains
- minimaxir 11y ago69 points, #3 on Hacker News, and no comments? :P This list would be much improved with descriptions for each dataset and indication of schema, as some of the datasets listed have very unfriendly schema. (e.g. the IMDB interfaces link) Kaggle's recently-released Public Datasets feature (https://www.kaggle.com/datasets https://www.kaggle.com/datasets) provides an interesting approach to presenting data and qualifying datasets by giving good examples of data robustness.
- qume 11y agoExactly my thought as soon as I looked at the lack of comments. How is it this community can debate some inconsequential nonsense, and there is no discussion here of how we get a consistent set of meta-data for these data sources. There are researchers both in academia and in the commercial world who would thrive if there were such a list with good consistent meta-data on how to interact with it. Disclosure: I work regularly with open datasets, and the effort it takes to work with each different set overshadows any effort on actual analysis.
- niels_olson 11y agoI'm building a dataset of images with annotations (svg-like as well as clinically relevant metadata). I would really like to know any papers or people who could help guide the effort. Any suggestions?
- developer2 11y agoI think it's really a political thing. How many of these open datasets come with a license/disclaimer that you are also allowed to rehost the data in its entirety for redistribution? It's one thing to standardize a format for open data, it's another thing to be able to make that standardized format available to everyone if you're repackaging someone else's data.
- rch 11y agoI'm working with a postgis corpus consisting of ACS, OES, CHSI, IPEDS, and many others. If you're in the same domain I'd love to chat sometime (email in profile).
- mistermann 11y ago> 69 points, #3 on Hacker News, and no comments? I expect this is because there is no ability on HN to differentiate (on articles and comments) between bookmark and upvote, most likely the majority of votes are for the purposes of bookmarking. Very often I want to upvote someone for a good comment, but I do that very sparingly now because I try to keep my upvoted comments list minimal so when I try to find something noteworthy I don't have to wade through pages of "liked" comments.
- dfc 11y agoI think you are drastically overestimating how many HN users are aware of the saved comments/stories feature.
- vosper 11y agoI've used HN for years, I've never even heard mention of saved comments/stories. What is this voodoo?
- cbd1984 11y agoMy saved comments: https://news.ycombinator.com/saved?id=cbd1984&comments=t https://news.ycombinator.com/saved?id=cbd1984&comments=t Your saved comments: https://news.ycombinator.com/saved?id=vosper&comments=t https://news.ycombinator.com/saved?id=vosper&comments=t My saved stories: https://news.ycombinator.com/saved?id=cbd1984 https://news.ycombinator.com/saved?id=cbd1984 Your saved stories: https://news.ycombinator.com/saved?id=vosper https://news.ycombinator.com/saved?id=vosper I have no idea if you'll be able to see mine or if I'll be able to see yours. We'll just have to find out together. Edited To Add: No. I can't, and you won't.
- tylercubell 11y agoEnigma.io is great for public data too.
- minimaxir 11y agoI took another look at the Enigma.io public datasets. Over 50% of all the public datasets are from the Federal Reserve Bank of St. Louis. Finance data is boring. :P Quandl (https://www.quandl.com/browse https://www.quandl.com/browse) is similar to Engima, except they got rid of all the fun datasets and added more finance/economic datasets. Hmrph.
- adkatrit 11y agoWhile the FRED data might be ~%50 of the datasets, most of those table are 200-4000 rows, it is not nearly %50 of the rows of data. the remaining 50% of datasets have a lot of gems quality in the breadth of data is important state-wide liquor license, corp reg. OSHA is great data. AMS shipping records. FDA adverse events. Oil and Gas well locations/production. Consolidated weather reports since 1800... im almost certainly forgetting some.
- vijayr 11y agoAre there sites that have datasets other than financial? (I also think financial data is too dry, boring)
- davecap1 11y agoSolveBio (my startup) has parsed, normalized, and indexed a bunch of the datasets listed under biology. Our goal is to make these kinds of datasets easier to access for programmers and non-programmers alike, similar to other some sites mentioned here (Enigma and Quandl) but for genomics. You can query and filter the data on the website or through one of our API clients: https://www.solvebio.com/library https://www.solvebio.com/library
- lap88 11y agoSounds like a good idea, especially the normalization part, but your site requires Java Script without even some basic functionality without it... nope.
- Spooky23 11y agoCheck out data.ny.gov Also nycopendata.socrata.com
- yzh 11y agoFor the complex network part, I think the collection missed this one: http://www.networkrepository.com/ http://www.networkrepository.com/ The site itself is a collection of several publicly available network datasets.
- clockwerx 11y agoI wish linkeddata.org or ckan installs weren't being reinvented here, but instead ckan supported pull requests or similar decentralized ways to publish new data sets
- rossj 11y agoIf you have any ideas/suggestions for how this might be implemented in CKAN, please do drop a mail to the list ( https://lists.okfn.org/mailman/listinfo/ckan-dev https://lists.okfn.org/mailman/listinfo/ckan-dev ) or add an issue at https://github.com/ckan/ideas-and-roadmap/issues https://github.com/ckan/ideas-and-roadmap/issues for discussion.
- jack9 11y agoI noticed no http://commoncrawl.org/ http://commoncrawl.org/ (oh no, naked domain!) or http://www.cochrane.org/ http://www.cochrane.org/ I don't quite understand the criteria for being included in the list since I think it's: https://groups.google.com/forum/#!forum/awesomepublicdatasets https://groups.google.com/forum/#!forum/awesomepublicdataset...
- lifeisstillgood 11y agoIs it too late to create a central registry of datasets - to aid discoverability. A voluntary system maintained by convention? Perhaps a distributed registration system ala DNS?
- Symbiote 11y agoWe have https://www.biodiversitycatalogue.org/ https://www.biodiversitycatalogue.org/ for biodiversity informatics APIs. A hackathon I attended made an API for registering, but I don't think it was deployed.
- mikedillion 11y agoSure, go ahead!
- dvcrn 11y agoI don't quite understand these awesome lists. From what I've seen it usually ends up being a way for creators to promote their stuff and for the list creator to have a big project with a few thousand stars in their profile. So when I did something in say, electron, I would go to the awesome-electron list and add it there for promotion sake. I couldn't find a usecase for these lists myself yet. There is no way to verify the quality of the product or the activity (stars for example? last commit date?). In one case I searched for aws adapters for a language, clicked on all links inside awesome-{{language}} just to find that all of them are inactive or a few days young. I ended up using something I found on google instead.
- detaro 11y agoIt really depends on how well they are maintained and how selective they are. (E.g. a long list with a few links for many categories is often more useful than one that lists everything that possibly fits in one category). And some basic information for each entry is really needed. GitHub offers a reasonable way to manage contributions to them, compared to many other solutions. It is easy to suggest/fix something for external contributors, but the owner can work as a gatekeeper. This is something many link aggregators or bookmarking sites lack. One example that isn't perfect, but I found interesting was https://github.com/Kickball/awesome-selfhosted https://github.com/Kickball/awesome-selfhosted It has a sentence about each project, license information, and tries to purge unmaintained projects.
- 7952 11y agoI think the benefit is as a source of inspiration. Finding data can require a lot of domain knowledge which students especially tend to lack. The problem is that the lists are always incomplete and other data may be more appropriate for a particular use case. Also, quality is more complex than just stars or activity. You need to know what the advantages or limitations are and how that applies to your project.
- elcapitan 11y agoWhen I started learning Golang last year I found the awesome-go list quite helpful. Not because I needed every single library mentioned, but because it quickly gave me an impression of how the ecosystem plays out and what typical ways to build stuff are. Because that's quite important in case you're not starting from an existing framework (.net or Rails etc).
- legulere 11y agoIt's strange that they put Wikidata under natural language.
- chestnut-tree 11y agoFor those in the UK, the available Government datasets are published on http://www.data.gov.uk http://www.data.gov.uk The datasets are not public domain, but licensed under the Open Government Licence (which allows you to use and adapt the data for commercial use). There's also the Global Open Data Index: a website that ranks countries by how much Government data is available as open datasets based on certain criteria. The current top spot is taken by Taiwan 1. Taiwan 2. UK 3. Denmark 4. Colombia 5. Finland 5. Australia 7. Uruguay 8. USA 8. Netherlands 10. Norway 10. France http://index.okfn.org/place/ http://index.okfn.org/place/
- psykovsky 11y agoYou mean Colombia?
- chestnut-tree 11y agoYes, sorry. Corrected :-)
- discardorama 11y agoThe author's notion of a "dataset" is weird. Under "Finance", there's a link to Google Finance page ( http://finance.google.com/ http://finance.google.com/ ). How is that a "dataset" ??
- patrickk 11y agoBetfair Historical Exchange Data requires you to have "100 Betfair points" which you acquire by gambling on their site. It's hardly an open dataset.