10 ms·
Free, Public Data Sets
- mindcrime 16y agoIf anyone is looking for more datasets, see: http://datasets.reddit.com http://datasets.reddit.com http://opendata.reddit.com http://opendata.reddit.com and http://www.quora.com/Where-can-I-get-large-datasets-open-to-the-public?q=dataset http://www.quora.com/Where-can-I-get-large-datasets-open-to-... for some good lists of available stuff.
- adw 16y agoWe've got quite a lot of public economic data: http://timetric.com/ http://timetric.com/. If you're up to something in the economic data space we'd love to talk. Happy to take this to email (andrew@timetric.com) if anyone's interested.
- hessenwolf 16y agoI looked at the site, and I see some data but I didn't find what I would have hoped for. I couldn't find yield curves, and historical exchange rates <i> up to <i/> today (available on the ecb site in xml format). Certainly I would have thought yield curves were a front page item. Things that would be very cool would be 1. financial statements in a database format. I know you can scrape this but I don't know if they are available legitimately? 2. Historial Implied volatilities and historical observed volatilities.
- adw 16y agohttp://timetric.com/dataset/exchange_rates_forex_europe/ http://timetric.com/dataset/exchange_rates_forex_europe/ for the exchange-rate data, at least.
- hessenwolf 16y agoOkay - it's there... Is it your site? Are you going to add yield curves?
- agentultra 16y agoWhat about http://ckan.org/ http://ckan.org/ ? The Comprehensive Knowledge Archive Network! Pretty sweet resource really.
- zipdog 16y agoThe wikipedia dump is great, but I've started using http://wiki.dbpedia.org/ http://wiki.dbpedia.org/ which has an API to query the dumps. Thanks for these, iisbum. I wish more public data was available in db, xml or similar structures - too often I find myself scraping government sites or pdfs to get the tables I need
- damoncali 16y agohttp://infochimps.com http://infochimps.com also has a bunch.
- sosuke 16y agoHeh, a day after he leaves HN he makes the first page. He will still be here whether he visits the site or not.
- balakk 16y agohttps://datamarket.azure.com/ https://datamarket.azure.com/ Some free, some paid.
- joubert 16y agoI have links to a few govt.-provided data sets at http://elev.at http://elev.at
- tszming 16y agoOpen Directory RDF Dump: http://rdf.dmoz.org/ http://rdf.dmoz.org/
- steveklabnik 16y agoDon't forget Stack Overflow! http://data.stackexchange.com/ http://data.stackexchange.com/
- llimllib 16y agohttp://www.gapminder.org/data/ http://www.gapminder.org/data/
- gtani 16y agohttp://www.kdnuggets.com/datasets/index.html http://www.kdnuggets.com/datasets/index.html http://lib.stat.cmu.edu/datasets/ http://lib.stat.cmu.edu/datasets/ http://datamob.org/ http://datamob.org/
- dmpayton 16y agoKinda surprised no one has mentioned Factual. I'm using some of their diabetes data for my side-startup. http://www.factual.com/ http://www.factual.com/
- casperc 16y agoTheir write that most the data is available for download. I can't find it anywhere though, only the various APIs. Have they remove the possibility of downloading the data?
- svag 16y agoThere is also the IMDB database in various format provided by IMDB itself here: http://www.imdb.com/interfaces http://www.imdb.com/interfaces Edit: Although the use of this database is not free, I believe for personal use is just fine to download and experiment...
- cstuder 16y agoAnd I recently discovered Google Refine, for cleaning up messy datasets. http://code.google.com/p/google-refine/ http://code.google.com/p/google-refine/
- LiveTheDream 16y agonée Freebase GridWorks http://blog.freebase.com/2010/11/10/google-refine-previously-freebase-gridworks-2-0-announced/ http://blog.freebase.com/2010/11/10/google-refine-previously...
- hvs 16y agoDon't forget the Lahman Baseball Database with information from 1871-2010 http://baseball1.com/statistics/ http://baseball1.com/statistics/
- jamwt 16y agoAnd, for very detailed play-by-play data for decades of games, check out retrosheet: http://www.retrosheet.org/game.htm http://www.retrosheet.org/game.htm
- seancron 16y agoHere's some more links to data sets: http://radar.oreilly.com/2010/03/open-data-pointers.html http://radar.oreilly.com/2010/03/open-data-pointers.html http://www.datawrangling.com/some-datasets-available-on-the-web http://www.datawrangling.com/some-datasets-available-on-the-... http://del.icio.us/pskomoroch/dataset http://del.icio.us/pskomoroch/dataset http://infochimps.com/collections/datamob http://infochimps.com/collections/datamob (and the other collections on the site) http://www.data.gov/ http://www.data.gov/
- wladimir 16y agoWow, useful stuff. This thread goes into my bookmarks.
- deleted 16y ago[deleted]
- Perceval 16y agoFor international relations data, Correlates of War hosts a number of data sets: http://www.correlatesofwar.org/Datasets.htm http://www.correlatesofwar.org/Datasets.htm
- jmtame 16y agoi've had trouble finding geographical boundaries on neighborhoods in U.S. cities (e.g. downtown areas and residential neighborhoods). anyone know where i can find this?
- ericwaller 16y agoIt's not in dump format, but you should take a look at simplegeo's (free) api: http://simplegeo.com/ http://simplegeo.com/
- kfranken 16y agoIt's not exactly neighborhoods, but the US Census TIGER database has block and blockgroup boundaries with associated demographic data. You could probably synthesize that into "neighborhood" definitions. http://www.census.gov/geo/www/tiger/tgrshp2010/tgrshp2010.html http://www.census.gov/geo/www/tiger/tgrshp2010/tgrshp2010.ht...
- timr 16y agohttp://www.zillow.com/howto/api/neighborhood-boundaries.htm http://www.zillow.com/howto/api/neighborhood-boundaries.htm
- _topher 16y agoThank you all for posting links and links to links to datasets, I have an unrelenting interest in data aggregation and machine learning, and didn't even know where to start. So helpful, and I am no longer stuck. :)
- fedd 16y agodo all of them have some uniformed api? that would be great, ideally. query and cache all of them on demand from your own app without additional programming. bookmarked and shared this thread.
- agbell 16y agoNon-Free Google data: http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2006T13 http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=... This data set, contributed by Google Inc., contains English word n-grams and their observed frequency counts.
- kaffeinecoma 16y agoThis is a real treasure to come across. I hope we'll keep seeing jacquesm's blog postings here. Anyone know of any publicly available song lyric databases?
- bravura 16y agoget.theinfo is the best way to find data sets. They are a bunch of data hoarders who can help you: http://groups.google.com/group/get-theinfo/?pli=1 http://groups.google.com/group/get-theinfo/?pli=1 I always ask there if I can't find what I'm looking for. Here are more and more data sets. These are general data sets. Email me if you have a specific data set in mind (e.g. web-as-corpus, spam, images, social, reviews, etc.). I have a big file of information. http://theinfo.org/ http://infochimps.org/datasets http://ckan.org [Comprehensive Knowledge Archive Network] http://www.datawrangling.com/some-datasets-available-on-the-web.html http://del.icio.us/pskomoroch/dataset http://www.reddit.com/r/datasets/ http://news.ycombinator.com/item?id=1242029 http://www.reddit.com/r/opendata http://www.trustlet.org/wiki/Repositories_of_datasets http://www.daniel-lemire.com/blog/data-for-data-mining/ http://www.quantlet.org/mdbase/ http://datamob.org/ http://freebase.com/ http://infochimp.info/ics/data/ripd/www-personal.umich.edu/~mejn/netdata/ http://www.archive-it.org/public/all_collections Large: http://www.ckan.net/tag/read/size-large http://www.diggingintodata.org/Repositories/tabid/167/Default.aspx Web as corpus: Good instructions: http://corpus.leeds.ac.uk/internet.html#description http://sslmit.unibo.it/~baroni/bootcat.html http://www.drni.de/wac-tk/index.php/Documentation etc. Email me if you need more http://cleaneval.sigwac.org.uk/ http://cleaneval.sigwac.org.uk/ http://liste.sslmit.unibo.it/pipermail/sigwac/2007-November/000041.html http://liste.sslmit.unibo.it/pipermail/sigwac/2007-November/... http://wacky.sslmit.unibo.it/doku.php?id= http://wacky.sslmit.unibo.it/doku.php?id= http://clic.cimec.unitn.it/marco/research.html http://clic.cimec.unitn.it/marco/research.html
- lkozma 16y agohttp://news.ycombinator.com/item?id=1493768 http://news.ycombinator.com/item?id=1493768
- LiveTheDream 16y agoI track datasets that I come across at http://www.delicious.com/tobym/dataset http://www.delicious.com/tobym/dataset
- WildUtah 16y agoDoes anybody have precinct-level election results for the USA? A set for recent elections would be great for public access redistricting apps that will become relevant this year.
- pwenzel 16y agoFor those interested in transit data, check out the GTFS Data Exchange, a directory of many agencies' scheduling and map data, following the Google Transit Feed Specification. http://www.gtfs-data-exchange.com/ http://www.gtfs-data-exchange.com/
- toisanji 16y agoanyone know of a dataset that has dates for when companies when companies registered or announced in the news? For example I would like to see the data hackernews was launched.
- nico_h 16y agohttp://www.naturalearthdata.com/ http://www.naturalearthdata.com/ From the website : Natural Earth is a public domain map dataset available at 1:10m, 1:50m, and 1:110 million scales as tightly integrated vector and raster data ...
- mcauser 16y agoHeaps of useful info: http://www.nationmaster.com http://www.nationmaster.com
- jcr 16y agoUnited Nations stats (lots of goodies) http://unstats.un.org http://unstats.un.org some free, some paid http://infochimps.com/ http://infochimps.com/ AIS Data (Marine Traffic) http://www.aishub.net/ http://www.aishub.net/ http://www.marinetraffic.com/ais/ http://www.marinetraffic.com/ais/ And there's a great list of sources on Quora http://www.quora.com/Where-can-I-get-large-datasets-open-to-the-public http://www.quora.com/Where-can-I-get-large-datasets-open-to-...
- eli 16y agoSome US Gov't data sites no one else mentioned: http://data.govloop.com/ http://data.govloop.com/ has data and lots of pointers to local government data. Also I'm surprised no one mentioned Carl Malamud's site: http://public.resource.org/ http://public.resource.org/ - Lots of US gov't and legal data in friendly formats.
- youknow 16y agoCIA World Factbook (demographics, geography, communications, government, economy, military stats of countries): https://www.cia.gov/library/publications/download/ https://www.cia.gov/library/publications/download/
- l0nwlf 16y agoOpenStreetMap data : http://wiki.openstreetmap.org/wiki/Planet.osm http://wiki.openstreetmap.org/wiki/Planet.osm Geonames : http://download.geonames.org/export/dump/ http://download.geonames.org/export/dump/ OS Open Data (UK Specific) : http://www.ordnancesurvey.co.uk/oswebsite/opendata/ http://www.ordnancesurvey.co.uk/oswebsite/opendata/
- nivertech 16y agoI looking for free public domain large high-resolution imaging datasets. Something like satellite imagery, medical imaging, semiconductor masks and wafers photos or CAD files, etc. Any pointers?
- brainid 16y agoHere are medical imaging datasets I am aware of: Neuroimaging (see http://www.nitrc.org http://www.nitrc.org for others) OASIS http://www.oasis-brains.org/ http://www.oasis-brains.org/ ADNI http://adni.loni.ucla.edu/ http://adni.loni.ucla.edu/ (huge dataset, requires application) OpenfMRI http://openfmri.org/ http://openfmri.org/ EEG http://eeg.pl/epi http://eeg.pl/epi Some other applications, example CT Colonography http://www.acrin.org/ http://www.acrin.org/
- random42 16y agoI'd prepared (based on other datasets) a smallish movie tweet dataset. You may find it useful, if working with tweets and/or reviews. https://github.com/mohitranka/TwitterSentimentCorpora https://github.com/mohitranka/TwitterSentimentCorpora
- random42 16y agoAnyplace I can find _small_ free web spam dataset? ( for commercial use, sorry :( ) All the datasets I found on www, are Huge (in double digit GBs..).