9 ms·

Google Dataset Search

1035 points by kmax12 8y ago

danso 8y ago
Very nice. Worth keeping in mind prior examples for comparison's sake. My favorites so far: - https://www.opendatanetwork.com https://www.opendatanetwork.com -- what I would call the "Google, for Socrata datasets" - https://public.enigma.com/ https://public.enigma.com/ -- One of the best collections of U.S. federal data, with good taxonomy and lots of useful options for refining a search, such as filtering by dataset size. - https://www.data.gov/ https://www.data.gov/ -- Not as useful as what most people would want -- e.g. unlike Enigma and Socrata, it's a directory of self-submitted (by the government) data sources, not one in which the data is stored/provided in a standardized way. But it's a pretty good listing, though not sure if it's much better than just using Google. - https://data.gov.uk/ https://data.gov.uk/ -- Better than the U.S. version in terms of usability and taxonomy.
- softwaredoug 8y ago
  Another example is Elsevier's Data Search project - https://datasearch.elsevier.com/ https://datasearch.elsevier.com/
- wiredfool 8y ago
  https://data.gov.ie https://data.gov.ie —- this is a repository of a bunch of the open data produced in the Irish public sector. (Disclosure, I work on this)
  - handelaar 8y ago
    I'm a particular fan of the boundary "dataset" you have that's a low-resolution TIFF file. (edit to add something more productive: the site is littered to the tune of at least 25% and maybe even a third with junk "data", all obviously added to get the number of records as high as possible, with no regard to whether that data is either useful to anybody, is machine-readable in any way at all, or even -- in the example above -- even qualifies as "data". Data.gov.ie would be moderately interesting if all the shit in it was removed.)
    - wiredfool 8y ago
      The quality of the datasets varies greatly depending on the source. Some work well, some, less so. There are data sources that are undergoing active development to harvest them more accurately. None of them were added to pump up the numbers. The biggest numbers bump recently was ca 1600 Met Eireann rainfall records datasets from all around the country, some of them daily rainfall dating back 60 years. (Spoiler, there’s a lot of rain) This is specifically a catalog of data sets, it doesn’t host the data except for previews, and even doing that is pretty complicated in all its glory.
      - handelaar 8y ago
        > None of them were added to pump up the numbers. Then kindly explain these https://data.gov.ie/advanced_search?query=&texttype=anywhere&org=adaptcentre&org=all-island-research-observatory&org=an-garda-siochana&org=benefacts&org=bord-iascaigh-mhara&org=central-applications-office&org=central-bank-of-ireland&org=central-statistics-office&org=coca-cola-zero-bikes&org=commissioners-of-irish-lights&org=commission-for-communications-regulation&org=commission-for-energy-regulation&org=companies-registration-office&org=cork-city-council&org=department-of-agriculture-food-and-the-marine&org=department-of-jobs-enterprise-and-innovation&org=department-of-children-and-youth-affairs&org=department-of-communications-climate-action-and-environment&org=department-of-arts-heritage-regional-rural-and-gaeltacht-affairs&org=department-of-education-and-skills&org=department-of-finance&org=department-of-foreign-affairs-and-trade&org=department-of-health&org=department-of-housing-planning-and-local-government&org=department-of-justice-equality&org=department-of-public-expenditure-and-reform&org=department-of-social-protection&org=department-of-transport-tourism-and-sport&org=donegal-county-council&org=dublin-city-business-improvement-district&org=dublin-city-council&org=dublinked&org=dublinkedie&org=dun-laoghaire-rathdown-county-council&org=ehealth-ireland&org=eirgrid&org=environmental-protection-agency&org=european-commission&org=eurostat&org=failte-ireland&org=fingal-county-council&org=fixyourstreetie&org=food-safety-authority-of-ireland&org=forfas&org=galway-city-council&org=galway-county-council&org=geological-survey-of-ireland&org=health-and-safety-authority&org=health-insurance-authority&org=health-products-regulatory-authority&org=health-protection-surveillance-centre&org=health-research-board&org=health-service-executive&org=heritage-council&org=higher-education-authority&org=oireachtas&org=injuriesboardie&org=irish-maritime-development-office&org=irish-naturalisation-and-immigration-service&org=irish-naval-service&org=irish-prison-service&org=irish-sports-council&org=isde&org=kildare-county-council&org=legal-aid-board&org=lgma&org=limerick-city-county-council&org=marine-institute&org=marine-renewable-energy-ireland-marei&org=mayo-county-council&org=meteireann&org=money-advice-and-budgeting-service&org=national-biodiversity-data-centre&org=national-educational-welfare-board&org=national-transport-authority&org=nui-galway&org=nui-maynooth&org=oecd&org=office-of-public-works&org=office-of-the-attorney-general&org=office-of-the-director-of-public-prosecutions&org=office-of-the-refugee-applications-commissioner&org=ordnance-survey-ireland&org=pharmaceutical-society-of-ireland&org=private-residential-tenancies-board&org=property-services-regulatory-authority&org=quality-and-qualifications-ireland&org=railway-procurement-agency&org=reception-and-integration-agency&org=revenue-commissioners&org=road-safety-authority&org=roscommon-county-council&org=royal-irish-academy&org=science-foundation-ireland&org=sea-fisheries-protection-authority&org=smartbay-ireland&org=solas&org=south-dublin-county-council&org=sport-ireland&org=statcentral&org=state-examinations-commission&org=sustainable-energy-authority-of-ireland&org=teagasc&org=the-adoption-authority-of-ireland&org=the-courts-service&org=the-national-cancer-registry&org=the-national-cancer-screening-service&org=the-national-treatment-purchase-fund&org=the-probation-service&org=transport-infrastructure-ireland&org=university-college-dublin&org=valuation-office&org=w3c_consortium&org=westmeath-county-council&org=wexford-county-council&licence=CC0-1.0&licence=CC-BY-4.0&licence=CC-BY-SA-4.0&licence=cc-by-sa&licence=cc-by-nc-nd&licence=GFDL-1.3-no-cover-texts-no-invariant-sections&licence=psi&licence=notspecified&licence=me-custom&licence=oir-psi&licence=ODC-BY-1.0&licence=ODbL-1.0&licence=ODC-PDDL-1.0&licence=OGL-UK-2.0&licence=other-at&licence=other-nc&licence=other-closed&licence=other-open&licence=other&licence=other-pd&theme=Agriculture&theme=Crime&theme=Economy&theme=Education+and+Sport&theme=Energy&theme=Environment&theme=Government&theme=Health&theme=Housing&theme=Society&theme=Science&theme=Towns&theme=Transport&format=TIFF&created_after=&created_before=&modified_after=&modified_before= https://data.gov.ie/advanced_search?query=&texttype=anywhere... Or these https://data.gov.ie/advanced_search?query=&texttype=anywhere&org=adaptcentre&org=all-island-research-observatory&org=an-garda-siochana&org=benefacts&org=bord-iascaigh-mhara&org=central-applications-office&org=central-bank-of-ireland&org=central-statistics-office&org=coca-cola-zero-bikes&org=commissioners-of-irish-lights&org=commission-for-communications-regulation&org=commission-for-energy-regulation&org=companies-registration-office&org=cork-city-council&org=department-of-agriculture-food-and-the-marine&org=department-of-jobs-enterprise-and-innovation&org=department-of-children-and-youth-affairs&org=department-of-communications-climate-action-and-environment&org=department-of-arts-heritage-regional-rural-and-gaeltacht-affairs&org=department-of-education-and-skills&org=department-of-finance&org=department-of-foreign-affairs-and-trade&org=department-of-health&org=department-of-housing-planning-and-local-government&org=department-of-justice-equality&org=department-of-public-expenditure-and-reform&org=department-of-social-protection&org=department-of-transport-tourism-and-sport&org=donegal-county-council&org=dublin-city-business-improvement-district&org=dublin-city-council&org=dublinked&org=dublinkedie&org=dun-laoghaire-rathdown-county-council&org=ehealth-ireland&org=eirgrid&org=environmental-protection-agency&org=european-commission&org=eurostat&org=failte-ireland&org=fingal-county-council&org=fixyourstreetie&org=food-safety-authority-of-ireland&org=forfas&org=galway-city-council&org=galway-county-council&org=geological-survey-of-ireland&org=health-and-safety-authority&org=health-insurance-authority&org=health-products-regulatory-authority&org=health-protection-surveillance-centre&org=health-research-board&org=health-service-executive&org=heritage-council&org=higher-education-authority&org=oireachtas&org=injuriesboardie&org=irish-maritime-development-office&org=irish-naturalisation-and-immigration-service&org=irish-naval-service&org=irish-prison-service&org=irish-sports-council&org=isde&org=kildare-county-council&org=legal-aid-board&org=lgma&org=limerick-city-county-council&org=marine-institute&org=marine-renewable-energy-ireland-marei&org=mayo-county-council&org=meteireann&org=money-advice-and-budgeting-service&org=national-biodiversity-data-centre&org=national-educational-welfare-board&org=national-transport-authority&org=nui-galway&org=nui-maynooth&org=oecd&org=office-of-public-works&org=office-of-the-attorney-general&org=office-of-the-director-of-public-prosecutions&org=office-of-the-refugee-applications-commissioner&org=ordnance-survey-ireland&org=pharmaceutical-society-of-ireland&org=private-residential-tenancies-board&org=property-services-regulatory-authority&org=quality-and-qualifications-ireland&org=railway-procurement-agency&org=reception-and-integration-agency&org=revenue-commissioners&org=road-safety-authority&org=roscommon-county-council&org=royal-irish-academy&org=science-foundation-ireland&org=sea-fisheries-protection-authority&org=smartbay-ireland&org=solas&org=south-dublin-county-council&org=sport-ireland&org=statcentral&org=state-examinations-commission&org=sustainable-energy-authority-of-ireland&org=teagasc&org=the-adoption-authority-of-ireland&org=the-courts-service&org=the-national-cancer-registry&org=the-national-cancer-screening-service&org=the-national-treatment-purchase-fund&org=the-probation-service&org=transport-infrastructure-ireland&org=university-college-dublin&org=valuation-office&org=w3c_consortium&org=westmeath-county-council&org=wexford-county-council&licence=CC0-1.0&licence=CC-BY-4.0&licence=CC-BY-SA-4.0&licence=cc-by-sa&licence=cc-by-nc-nd&licence=GFDL-1.3-no-cover-texts-no-invariant-sections&licence=psi&licence=notspecified&licence=me-custom&licence=oir-psi&licence=ODC-BY-1.0&licence=ODbL-1.0&licence=ODC-PDDL-1.0&licence=OGL-UK-2.0&licence=other-at&licence=other-nc&licence=other-closed&licence=other-open&licence=other&licence=other-pd&theme=Agriculture&theme=Crime&theme=Economy&theme=Education+and+Sport&theme=Energy&theme=Environment&theme=Government&theme=Health&theme=Housing&theme=Society&theme=Science&theme=Towns&theme=Transport&format=TIFF&created_after=&created_before=&modified_after=&modified_before= https://data.gov.ie/advanced_search?query=&texttype=anywhere... (I'd show only the PDF-only ones but your search doesn't work.) Oh and look two of these also contain no machine-readable data whatsoever https://data.gov.ie/advanced_search?query=&texttype=anywhere&org=adaptcentre&org=all-island-research-observatory&org=an-garda-siochana&org=benefacts&org=bord-iascaigh-mhara&org=central-applications-office&org=central-bank-of-ireland&org=central-statistics-office&org=coca-cola-zero-bikes&org=commissioners-of-irish-lights&org=commission-for-communications-regulation&org=commission-for-energy-regulation&org=companies-registration-office&org=cork-city-council&org=department-of-agriculture-food-and-the-marine&org=department-of-jobs-enterprise-and-innovation&org=department-of-children-and-youth-affairs&org=department-of-communications-climate-action-and-environment&org=department-of-arts-heritage-regional-rural-and-gaeltacht-affairs&org=department-of-education-and-skills&org=department-of-finance&org=department-of-foreign-affairs-and-trade&org=department-of-health&org=department-of-housing-planning-and-local-government&org=department-of-justice-equality&org=department-of-public-expenditure-and-reform&org=department-of-social-protection&org=department-of-transport-tourism-and-sport&org=donegal-county-council&org=dublin-city-business-improvement-district&org=dublin-city-council&org=dublinked&org=dublinkedie&org=dun-laoghaire-rathdown-county-council&org=ehealth-ireland&org=eirgrid&org=environmental-protection-agency&org=european-commission&org=eurostat&org=failte-ireland&org=fingal-county-council&org=fixyourstreetie&org=food-safety-authority-of-ireland&org=forfas&org=galway-city-council&org=galway-county-council&org=geological-survey-of-ireland&org=health-and-safety-authority&org=health-insurance-authority&org=health-products-regulatory-authority&org=health-protection-surveillance-centre&org=health-research-board&org=health-service-executive&org=heritage-council&org=higher-education-authority&org=oireachtas&org=injuriesboardie&org=irish-maritime-development-office&org=irish-naturalisation-and-immigration-service&org=irish-naval-service&org=irish-prison-service&org=irish-sports-council&org=isde&org=kildare-county-council&org=legal-aid-board&org=lgma&org=limerick-city-county-council&org=marine-institute&org=marine-renewable-energy-ireland-marei&org=mayo-county-council&org=meteireann&org=money-advice-and-budgeting-service&org=national-biodiversity-data-centre&org=national-educational-welfare-board&org=national-transport-authority&org=nui-galway&org=nui-maynooth&org=oecd&org=office-of-public-works&org=office-of-the-attorney-general&org=office-of-the-director-of-public-prosecutions&org=office-of-the-refugee-applications-commissioner&org=ordnance-survey-ireland&org=pharmaceutical-society-of-ireland&org=private-residential-tenancies-board&org=property-services-regulatory-authority&org=quality-and-qualifications-ireland&org=railway-procurement-agency&org=reception-and-integration-agency&org=revenue-commissioners&org=road-safety-authority&org=roscommon-county-council&org=royal-irish-academy&org=science-foundation-ireland&org=sea-fisheries-protection-authority&org=smartbay-ireland&org=solas&org=south-dublin-county-council&org=sport-ireland&org=statcentral&org=state-examinations-commission&org=sustainable-energy-authority-of-ireland&org=teagasc&org=the-adoption-authority-of-ireland&org=the-courts-service&org=the-national-cancer-registry&org=the-national-cancer-screening-service&org=the-national-treatment-purchase-fund&org=the-probation-service&org=transport-infrastructure-ireland&org=university-college-dublin&org=valuation-office&org=w3c_consortium&org=westmeath-county-council&org=wexford-county-council&licence=CC0-1.0&licence=CC-BY-4.0&licence=CC-BY-SA-4.0&licence=cc-by-sa&licence=cc-by-nc-nd&licence=GFDL-1.3-no-cover-texts-no-invariant-sections&licence=psi&licence=notspecified&licence=me-custom&licence=oir-psi&licence=ODC-BY-1.0&licence=ODbL-1.0&licence=ODC-PDDL-1.0&licence=OGL-UK-2.0&licence=other-at&licence=other-nc&licence=other-closed&licence=other-open&licence=other&licence=other-pd&theme=Agriculture&theme=Crime&theme=Economy&theme=Education+and+Sport&theme=Energy&theme=Environment&theme=Government&theme=Health&theme=Housing&theme=Society&theme=Science&theme=Towns&theme=Transport&format=TIFF&created_after=&created_before=&modified_after=&modified_before= https://data.gov.ie/advanced_search?query=&texttype=anywhere...
  - eruci 8y ago
    Are you going to release any street address datasets at some point?
    - wiredfool 8y ago
      I believe that the eircode dataset is one of the most highly requested sources, but it’s a private for profit database. If you are aware of such a dataset that a public body is hosting, then it would certainly be something to include. Convincing (and helping) the public bodies to publish their data is still a big task.
      - disgruntledphd2 8y ago
        I believe that An Post built a similar system many moons ago, and they should presumably be more amenable to open-sourcing. The handling of the whole Eircode thing makes my blood boil, to be honest.
  - fnord123 8y ago
    Playing around, a lot of countries use this url scheme: https://data.gov.be https://data.gov.be http://data.gov.ro/ http://data.gov.ro/ https://www.dati.gov.it/ https://www.dati.gov.it/ (note, Italy redirects data. to dati.) https://data.gov.pl/ https://data.gov.pl/ https://data.gov.za/ https://data.gov.za/ (South Africa's has a cert problem) https://data.gov.in https://data.gov.in https://data.gov.au/ https://data.gov.au/ https://datos.gob.mx/ https://datos.gob.mx/
    - snaily 8y ago
      https://data.gov.sg/ https://data.gov.sg/ (Singapore)
    - vquemener 8y ago
      https://www.data.gouv.fr https://www.data.gouv.fr (France)
    - nonnon 8y ago
      https://www.data.gv.at/ https://www.data.gv.at/ (Austria)
    - mnx 8y ago
      although data.gov.pl exists, and even presents a valid cert, it has no content. The place with the data is: https://danepubliczne.gov.pl/en/# https://danepubliczne.gov.pl/en/# (website is being terribly slow for me right now)
    - Moghammed 8y ago
      https://data.overheid.nl/ https://data.overheid.nl/
    - divbyzer0 8y ago
      https://data.gov.ie/ https://data.gov.ie/
- alphakappa 8y ago
  Also, https://data.gov.in https://data.gov.in -- Lots of self-submitted datasets, and includes an API. Usability is average, much like the other government data sources.
- philipashlock 8y ago
  @danso thanks for the feedback on data.gov. I'm part of the small (3 person) team that helps to manage it. If you have a moment to chat I'll see if I can reach out to you to see if you'd be interested in participating in some more in-depth user research in the future. Folks can also always leave feedback via email, github, twitter, and other means - https://www.data.gov/contact https://www.data.gov/contact The Federal Data Strategy will also be opening up for comments again in October - https://strategy.data.gov/feedback/ https://strategy.data.gov/feedback/ Data.gov and Federal agencies use the same metadata standard (DCAT) that Google Dataset Search is using so much of our metadata is also being syndicated there.
  - thebooglebooski 8y ago
    I think the biggest blocker with using publicly available datasets is stale data. If you, or anyone else who aggregates these datasets could make it EASY to find the FREQUENCY of updates, rather than just the LAST UPDATED timestamp, it'd incentivize people to consume APIs more. I realize having a snapshot from 2014 is better than what was publicly available before. But I feel no one's really talked about why they would or wouldn't use particular data.
    - izyda 8y ago
      I think this is exactly correct. Frequency of updates (and clear documentation of the lag relationship between when data is reported and for what period data is applicable too) is often missing or hard to find. The value of increasing the cadence of updates should also not be understated! A lot of public dataset report on annual frequencies with more than a quarter of delay... Although this is a different issue altogether that has more to do with the processes of the reporting agency.
      - chetanbhasin 8y ago
        Yes, it's interesting how much difference the data about data management can make in people's engagement with the platform.
  - 8y ago
- Cd00d 8y ago
  I really like NYC Open Data: https://opendata.cityofnewyork.us/ https://opendata.cityofnewyork.us/
- Untit1ed 8y ago
  Blatantly self-promoting, but we (in Australia) are trying to develop a general solution for better open data search over at https://search.data.gov.au https://search.data.gov.au - solution is all open source at https://github.com/magda-io/magda https://github.com/magda-io/magda.
- rufuspollock 8y ago
  Interesting. There have been a lot of attempts at "meta data portals" that search across portals. Most of them have struggled. At Open Knowledge we built a really early one called opendatasearch.org in 2011/2012 - now defunct - and were involved in the first version of the pan EU open data portal. We also had the original https://ckan.net/ https://ckan.net/ (and subsites) which is now https://datahub.io/ https://datahub.io/ and has become much more focused on quality data and data deployment. [Disclosure: I was/am involved in many of these projects] The challenge, as others have mentioned, is that data quality is very variable and searching for datasets is complicated (think of software as an analogy - searching for good code libraries is a bit of an art). I imagine Google are trying this out before making datasets another "special type" of search result -- after all you can already search google for datasets. In addition, Google are already Google so including datasets will have a level of comprehensiveness and exposure you struggle with elsewhere (part of the power of monopoly in a sense!). PS: for those looking for data gov sites https://dataportals.org/ https://dataportals.org/ has most of these.
minimaxir 8y ago
Interesting vertical integration for Kaggle Datasets: https://www.kaggle.com/datasets https://www.kaggle.com/datasets
- rufuspollock 8y ago
  Note Kaggle recently adopted the FrictionlessData.io Data Package specs: https://github.com/Kaggle/kaggle-api/wiki/Dataset-Metadata https://github.com/Kaggle/kaggle-api/wiki/Dataset-Metadata
afettere 8y ago
https://data.urbanfootprint.com/ https://data.urbanfootprint.com/ - browse thousands of environmental, social, transportation, and land use datasets. (disclosure - I work there)
Dangeranger 8y ago
Nice. I'm glad Google is making it easier to find public data sets. I wish that these could be filtered by format, so that you could narrow them to CSV, XML, JSON, KML, etc. Another nice resource that I've used in the past is 'toddmotto/public-apis' on Github [0]. In the end I would prefer all public data sets to be available over the DAT protocol [1] instead of being hosted only on government or organization websites. A lot of climate data previously made available by the EPA was taken down, and only saved by efforts of volunteers.[2] [0] https://github.com/toddmotto/public-apis https://github.com/toddmotto/public-apis [1] https://datproject.org/ https://datproject.org/ [2] https://www.wired.com/2017/01/rogue-scientists-race-save-climate-data-trump/ https://www.wired.com/2017/01/rogue-scientists-race-save-cli...
- neuromantik8086 8y ago
  Dat's pretty cool, but it's not the only game out there. The efforts of git-annex/Datalad [0], Academic Torrents [1], Quilt[2], DVC[3], and Pachyderm [4] are also notable in this space. My hopes are broader in the sense that I just hope that dataset versioning happens in the first place. [0] https://www.datalad.org/ https://www.datalad.org/ [1] http://academictorrents.com/ http://academictorrents.com/ [2] https://quiltdata.com/ https://quiltdata.com/ [3] https://dvc.org/ https://dvc.org/ [4] http://www.pachyderm.io/ http://www.pachyderm.io/
  - rgardaphe 8y ago
    [5] https://qri.io https://qri.io (we're tackling the dataset versioning problem head on.
- sixdimensional 8y ago
  Yeah, if they had added an attribute for formats available for each data set, and then added a filter by type to the search (e.g. "type:csv") or similar, it would be great. Sometimes you really want a specific format for a dataset.
  - matt4077 8y ago
ericand 8y ago
What's up with this domain? Is there anything else under toolbox.google.com?
- deleted 8y ago
  [deleted]
- amadeuspzs 8y ago
  Here you go: http://bfy.tw/JkG7 http://bfy.tw/JkG7
  - dang 8y ago
    Please don't do this here.
jonbaer 8y ago
Using Google to search Google ... https://toolbox.google.com/datasetsearch/search?query=site%3Agoogle.com https://toolbox.google.com/datasetsearch/search?query=site%3...
neuromantik8086 8y ago
Looks like academic institutional repositories and figshare are doing the heavy lifting here. It's still neat to see Google aggregate everything, but it's not that different from what they do with other services relying on these sources already, and is largely dependent upon how rich these upstream sources are in the first place.
earth2mars 8y ago
It would be good to know who sponsored the data research so that everyone can make self biased decision to use/trust the data.
MrEfficiency 8y ago
What is the best way for a website to format data to the public? I already have my presentation, but I can also provide it as a .xls, .csv, sql, or html table. What would be best to help programmers/data scientists use my data?
- polkapolka 8y ago
  Csv and a permitive license like CC0. Provide good meta data to make provenance easier. Mark up your dataset with schema.org.
- combatentropy 8y ago
  I vote CSV. - simple - lightweight - open It's easy for the consumer of your data to convert a CSV to whatever format they need. - spreadsheet, for personal analysis - SQL database, for industrial-strength analysis - HTML, for pretty output to their users
  - lolive 8y ago
    Oh, and don't forget that CSV is a rigorous data format, with many tricks. Don't just append some text together, separated by commas, and call it CSV. Instead, use a dedicated lib to create the CSV for you.
- heinrichhartman 8y ago
  1) Put it on figshare or github. Your website might be dead next year. Those services have a better chance of survival. 2) CSV, JSON works fine if your dataset is just a few numbers and strings. GitHub will preview csv files as HTML tables. 3) If you need the efficiency of binary data and more robust data containers I would look into - Parquet https://parquet.apache.org/documentation/latest/ https://parquet.apache.org/documentation/latest/ and - Arvo http://avro.apache.org/ http://avro.apache.org/ http://blog.cloudera.com/blog/2009/11/avro-a-new-format-for-data-interchange/ http://blog.cloudera.com/blog/2009/11/avro-a-new-format-for-... 4) DataScientists work with R/Pandas "DataFrames". If you are familiar with either one, import the data into a data frame and use an export method to do the serialization for you: https://pandas.pydata.org/pandas-docs/stable/api.html#id12 https://pandas.pydata.org/pandas-docs/stable/api.html#id12
michaelmior 8y ago
Of note is the link below which indicates how you can have your dataset indexed. https://developers.google.com/search/docs/data-types/dataset https://developers.google.com/search/docs/data-types/dataset
xaranke 8y ago
Placeholder comment to calculate the time before Google inevitably shuts this down.
heinrichhartman 8y ago
The state of data sharing seems to be still quite sad. * Hosting problems. The first link I tried was already broken. * Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?). * Update problems. Many data-sets change over time (e.g. GDP). How can I subscribe to updates? "git pull" would be nice. * Provenance problems. I want to know who put which record into the dataset, when and why? "git log" would be nice. * Presentation problems. (This is OK sometimes) I necessarily want to download 5Gb file before I looked into it. The first few rows of the dataset should be presented on the page, with information about it. Wrote down a few more thoughts a while ago here: https://github.com/HeinrichHartmann/data-sharing#in-the-ideal-world https://github.com/HeinrichHartmann/data-sharing#in-the-idea... Approaches I have seen so far in the wild: * figshare.com -- Addresses Hosting and Presentation. * https://quiltdata.com/ https://quiltdata.com/ -- (!) looks great. Still exploring. * github.com -- works fine for small datasets (<1GB) * packaging (yum, pkg, pip) -- (?) Not sure if that works, but at least they solve: Hosting, Update, Provenance. This seems to be a wide open problem to me.
- rgardaphe 8y ago
  Totally agree! At Qri (https://qri.io https://qri.io) we're working on many of these problems together - hosting, formatting (interoperability), provenance and sync. It's an open source project - we'd love to have your feedback as we design it!
  - sixdimensional 8y ago
    This is cool and is a perfect case for IPFS for public datasets. I've not heard of it before, though, and I think naming / branding is something that makes finding these things / building momentum more difficult. For example, someone else mentioned engima.com. I would have no idea that is related to data sources / sets unless I knew what it was. Certainly wish you the best of luck though and will keep an eye on Qri! Cool project!
rebel1 8y ago
Interestingly enough, I built and released something very similar [1] about a month ago using a Google Custom Search Engine. Here is the Show HN for it: https://news.ycombinator.com/item?id=17789119 https://news.ycombinator.com/item?id=17789119 [1] https://databasd.com/ https://databasd.com/
spoiledtechie 8y ago
Can we list some good data searches to suggest below this comment? I would love to see some cool data we might be able to use.
sbr464 8y ago
This is nice. I'm working on a similar open source project that is releasing soon called DataLibrary[0] It goes further by bringing this kind of data together into a single API, converting/cleaning into a similar schema where possible. A small write up can be found on github [1]. Any feedback/ideas would be appreciated! [0] https://www.datalibrary.com https://www.datalibrary.com (not online currently) [1] https://github.com/reactual/datalibrary/blob/master/README.md https://github.com/reactual/datalibrary/blob/master/README.m...
madisonmay 8y ago
I also very much like https://www.figure-eight.com/data-for-everyone/ https://www.figure-eight.com/data-for-everyone/. It's not optimized for search but it's an excellent repository of high quality datasets.
smartvlad 8y ago
Look at https://knoema.com https://knoema.com which positions itself as a search engine for data with more than 2.5 billion time series available. They provide both visual data discovery through search and navigation as well as API access through Python, R etc.
ToFab123 8y ago
Pretty lame that it doesn't work in Edge.
choicenotchance 8y ago
Most of them are paid.
sytelus 8y ago
Looks like this doesn’t crawl one of the most important source for data: academic torrents. For example, I searched for ilsvrc lmdb This should have found imagenet data in lmdb format available somewhere but it returned no results.
frabcus 8y ago
Nice to see Google trying this again! It's one of those areas they have long attempts at involvement in - e.g. Google Public Data Explorer which never quite reached it's potential, and Freebase which although flawed was good and was shut down after Google acquired it. I like that this is search based! The web is still the best place to publish data - in fact in my view normal Google search is still by far the best way to find datasets, even though it isn't directly designed for that. There's a link from the about page of Google Dataset Search to this help for webmasters on how to mark up content for it - although it is a bit odd, mainly showing how to mark a dataset with a DOI (so good for academics certainly!): https://productforums.google.com/forum/#!topic/webmasters/nPq4BW6iPIA https://productforums.google.com/forum/#!topic/webmasters/nP... Just metadata about data feels like a very niche thing to search to me - I'm still not convinced anyone will maintain the metadata well enough to help. Possibly will work in particular domains. Does Dataset Search have some way to search column headings, types or content (of CSV, Excel, JSON etc)? I can imagine a load of operators that would make that really powerful for finding badly meta-marked up datasets deep in the web. Would seem like the obvious extra thing a dataset search would do. Also previews please!!! Just nicely render the fist ten rows of common formats - CSV and Excel to begin with. What part of Google is doing this?
a2x 8y ago
Google is so damn bad on UX. An excellent example of interaction of unlimited ressources and priorities.
nterpo 8y ago
You may want to try [https://data.opendatasoft.com](https://data.opendatasoft.com) https://data.opendatasoft.com](https://data.opendatasoft.com... -- thousands of datasets available through the same API, usable online, no download required.
xcubic 8y ago
Add this one: https://dados.gov.pt/ https://dados.gov.pt/ (Portugal)
deleted 8y ago
[deleted]
jwillmer 8y ago
Since we have already a nice reference list of open data portals by country I like to add the German portal: https://www.govdata.de/ https://www.govdata.de/
deleted 8y ago
[deleted]
lmy86263 8y ago
it will be helpful for somebody who major in AI
sprague 8y ago
Then there's this $500K just awarded by the NSF to build a "Google for data sets". I wonder if, before making these sorts of grants, the NSF looks at what Google and other and other companies are already (or likely) doing. https://www.lehigh.edu/engineering/news/faculty/2018/20180820-davison-heflin-jia-dataset-search-engine-nsf-award.html https://www.lehigh.edu/engineering/news/faculty/2018/2018082...
MR4D 8y ago
Maybe I’m missing something but this strikes me as underwhelming. To the point of something that I could do as opposed to something that the firm that created maps and gmail could do. Is it just me?
ece 8y ago
Nothing for FRED https://toolbox.google.com/datasetsearch/search?query=site%3Afred.stlouisfed.org https://toolbox.google.com/datasetsearch/search?query=site%3... hmm...

1035 points by kmax12 8y ago

Very nice. Worth keeping in mind prior examples for comparison's sake. My favorites so far: - https://www.opendatanetwork.com https://www.opendatanetwork.com -- what I would call the "Google, for Socrata datasets" - https://public.enigma.com/ https://public.enigma.com/ -- One of the best collections of U.S. federal data, with good taxonomy and lots of useful options for refining a search, such as filtering by dataset size. - https://www.data.gov/ https://www.data.gov/ -- Not as useful as what most people would want -- e.g. unlike Enigma and Socrata, it's a directory of self-submitted (by the government) data sources, not one in which the data is stored/provided in a standardized way. But it's a pretty good listing, though not sure if it's much better than just using Google. - https://data.gov.uk/ https://data.gov.uk/ -- Better than the U.S. version in terms of usability and taxonomy.

softwaredoug 8y ago
Another example is Elsevier's Data Search project - https://datasearch.elsevier.com/ https://datasearch.elsevier.com/
wiredfool 8y ago
https://data.gov.ie https://data.gov.ie —- this is a repository of a bunch of the open data produced in the Irish public sector. (Disclosure, I work on this)
- handelaar 8y ago
  I'm a particular fan of the boundary "dataset" you have that's a low-resolution TIFF file. (edit to add something more productive: the site is littered to the tune of at least 25% and maybe even a third with junk "data", all obviously added to get the number of records as high as possible, with no regard to whether that data is either useful to anybody, is machine-readable in any way at all, or even -- in the example above -- even qualifies as "data". Data.gov.ie would be moderately interesting if all the shit in it was removed.)
  - wiredfool 8y ago
    The quality of the datasets varies greatly depending on the source. Some work well, some, less so. There are data sources that are undergoing active development to harvest them more accurately. None of them were added to pump up the numbers. The biggest numbers bump recently was ca 1600 Met Eireann rainfall records datasets from all around the country, some of them daily rainfall dating back 60 years. (Spoiler, there’s a lot of rain) This is specifically a catalog of data sets, it doesn’t host the data except for previews, and even doing that is pretty complicated in all its glory.
    - handelaar 8y ago
      > None of them were added to pump up the numbers. Then kindly explain these https://data.gov.ie/advanced_search?query=&texttype=anywhere&org=adaptcentre&org=all-island-research-observatory&org=an-garda-siochana&org=benefacts&org=bord-iascaigh-mhara&org=central-applications-office&org=central-bank-of-ireland&org=central-statistics-office&org=coca-cola-zero-bikes&org=commissioners-of-irish-lights&org=commission-for-communications-regulation&org=commission-for-energy-regulation&org=companies-registration-office&org=cork-city-council&org=department-of-agriculture-food-and-the-marine&org=department-of-jobs-enterprise-and-innovation&org=department-of-children-and-youth-affairs&org=department-of-communications-climate-action-and-environment&org=department-of-arts-heritage-regional-rural-and-gaeltacht-affairs&org=department-of-education-and-skills&org=department-of-finance&org=department-of-foreign-affairs-and-trade&org=department-of-health&org=department-of-housing-planning-and-local-government&org=department-of-justice-equality&org=department-of-public-expenditure-and-reform&org=department-of-social-protection&org=department-of-transport-tourism-and-sport&org=donegal-county-council&org=dublin-city-business-improvement-district&org=dublin-city-council&org=dublinked&org=dublinkedie&org=dun-laoghaire-rathdown-county-council&org=ehealth-ireland&org=eirgrid&org=environmental-protection-agency&org=european-commission&org=eurostat&org=failte-ireland&org=fingal-county-council&org=fixyourstreetie&org=food-safety-authority-of-ireland&org=forfas&org=galway-city-council&org=galway-county-council&org=geological-survey-of-ireland&org=health-and-safety-authority&org=health-insurance-authority&org=health-products-regulatory-authority&org=health-protection-surveillance-centre&org=health-research-board&org=health-service-executive&org=heritage-council&org=higher-education-authority&org=oireachtas&org=injuriesboardie&org=irish-maritime-development-office&org=irish-naturalisation-and-immigration-service&org=irish-naval-service&org=irish-prison-service&org=irish-sports-council&org=isde&org=kildare-county-council&org=legal-aid-board&org=lgma&org=limerick-city-county-council&org=marine-institute&org=marine-renewable-energy-ireland-marei&org=mayo-county-council&org=meteireann&org=money-advice-and-budgeting-service&org=national-biodiversity-data-centre&org=national-educational-welfare-board&org=national-transport-authority&org=nui-galway&org=nui-maynooth&org=oecd&org=office-of-public-works&org=office-of-the-attorney-general&org=office-of-the-director-of-public-prosecutions&org=office-of-the-refugee-applications-commissioner&org=ordnance-survey-ireland&org=pharmaceutical-society-of-ireland&org=private-residential-tenancies-board&org=property-services-regulatory-authority&org=quality-and-qualifications-ireland&org=railway-procurement-agency&org=reception-and-integration-agency&org=revenue-commissioners&org=road-safety-authority&org=roscommon-county-council&org=royal-irish-academy&org=science-foundation-ireland&org=sea-fisheries-protection-authority&org=smartbay-ireland&org=solas&org=south-dublin-county-council&org=sport-ireland&org=statcentral&org=state-examinations-commission&org=sustainable-energy-authority-of-ireland&org=teagasc&org=the-adoption-authority-of-ireland&org=the-courts-service&org=the-national-cancer-registry&org=the-national-cancer-screening-service&org=the-national-treatment-purchase-fund&org=the-probation-service&org=transport-infrastructure-ireland&org=university-college-dublin&org=valuation-office&org=w3c_consortium&org=westmeath-county-council&org=wexford-county-council&licence=CC0-1.0&licence=CC-BY-4.0&licence=CC-BY-SA-4.0&licence=cc-by-sa&licence=cc-by-nc-nd&licence=GFDL-1.3-no-cover-texts-no-invariant-sections&licence=psi&licence=notspecified&licence=me-custom&licence=oir-psi&licence=ODC-BY-1.0&licence=ODbL-1.0&licence=ODC-PDDL-1.0&licence=OGL-UK-2.0&licence=other-at&licence=other-nc&licence=other-closed&licence=other-open&licence=other&licence=other-pd&theme=Agriculture&theme=Crime&theme=Economy&theme=Education+and+Sport&theme=Energy&theme=Environment&theme=Government&theme=Health&theme=Housing&theme=Society&theme=Science&theme=Towns&theme=Transport&format=TIFF&created_after=&created_before=&modified_after=&modified_before= https://data.gov.ie/advanced_search?query=&texttype=anywhere... Or these https://data.gov.ie/advanced_search?query=&texttype=anywhere&org=adaptcentre&org=all-island-research-observatory&org=an-garda-siochana&org=benefacts&org=bord-iascaigh-mhara&org=central-applications-office&org=central-bank-of-ireland&org=central-statistics-office&org=coca-cola-zero-bikes&org=commissioners-of-irish-lights&org=commission-for-communications-regulation&org=commission-for-energy-regulation&org=companies-registration-office&org=cork-city-council&org=department-of-agriculture-food-and-the-marine&org=department-of-jobs-enterprise-and-innovation&org=department-of-children-and-youth-affairs&org=department-of-communications-climate-action-and-environment&org=department-of-arts-heritage-regional-rural-and-gaeltacht-affairs&org=department-of-education-and-skills&org=department-of-finance&org=department-of-foreign-affairs-and-trade&org=department-of-health&org=department-of-housing-planning-and-local-government&org=department-of-justice-equality&org=department-of-public-expenditure-and-reform&org=department-of-social-protection&org=department-of-transport-tourism-and-sport&org=donegal-county-council&org=dublin-city-business-improvement-district&org=dublin-city-council&org=dublinked&org=dublinkedie&org=dun-laoghaire-rathdown-county-council&org=ehealth-ireland&org=eirgrid&org=environmental-protection-agency&org=european-commission&org=eurostat&org=failte-ireland&org=fingal-county-council&org=fixyourstreetie&org=food-safety-authority-of-ireland&org=forfas&org=galway-city-council&org=galway-county-council&org=geological-survey-of-ireland&org=health-and-safety-authority&org=health-insurance-authority&org=health-products-regulatory-authority&org=health-protection-surveillance-centre&org=health-research-board&org=health-service-executive&org=heritage-council&org=higher-education-authority&org=oireachtas&org=injuriesboardie&org=irish-maritime-development-office&org=irish-naturalisation-and-immigration-service&org=irish-naval-service&org=irish-prison-service&org=irish-sports-council&org=isde&org=kildare-county-council&org=legal-aid-board&org=lgma&org=limerick-city-county-council&org=marine-institute&org=marine-renewable-energy-ireland-marei&org=mayo-county-council&org=meteireann&org=money-advice-and-budgeting-service&org=national-biodiversity-data-centre&org=national-educational-welfare-board&org=national-transport-authority&org=nui-galway&org=nui-maynooth&org=oecd&org=office-of-public-works&org=office-of-the-attorney-general&org=office-of-the-director-of-public-prosecutions&org=office-of-the-refugee-applications-commissioner&org=ordnance-survey-ireland&org=pharmaceutical-society-of-ireland&org=private-residential-tenancies-board&org=property-services-regulatory-authority&org=quality-and-qualifications-ireland&org=railway-procurement-agency&org=reception-and-integration-agency&org=revenue-commissioners&org=road-safety-authority&org=roscommon-county-council&org=royal-irish-academy&org=science-foundation-ireland&org=sea-fisheries-protection-authority&org=smartbay-ireland&org=solas&org=south-dublin-county-council&org=sport-ireland&org=statcentral&org=state-examinations-commission&org=sustainable-energy-authority-of-ireland&org=teagasc&org=the-adoption-authority-of-ireland&org=the-courts-service&org=the-national-cancer-registry&org=the-national-cancer-screening-service&org=the-national-treatment-purchase-fund&org=the-probation-service&org=transport-infrastructure-ireland&org=university-college-dublin&org=valuation-office&org=w3c_consortium&org=westmeath-county-council&org=wexford-county-council&licence=CC0-1.0&licence=CC-BY-4.0&licence=CC-BY-SA-4.0&licence=cc-by-sa&licence=cc-by-nc-nd&licence=GFDL-1.3-no-cover-texts-no-invariant-sections&licence=psi&licence=notspecified&licence=me-custom&licence=oir-psi&licence=ODC-BY-1.0&licence=ODbL-1.0&licence=ODC-PDDL-1.0&licence=OGL-UK-2.0&licence=other-at&licence=other-nc&licence=other-closed&licence=other-open&licence=other&licence=other-pd&theme=Agriculture&theme=Crime&theme=Economy&theme=Education+and+Sport&theme=Energy&theme=Environment&theme=Government&theme=Health&theme=Housing&theme=Society&theme=Science&theme=Towns&theme=Transport&format=TIFF&created_after=&created_before=&modified_after=&modified_before= https://data.gov.ie/advanced_search?query=&texttype=anywhere... (I'd show only the PDF-only ones but your search doesn't work.) Oh and look two of these also contain no machine-readable data whatsoever https://data.gov.ie/advanced_search?query=&texttype=anywhere&org=adaptcentre&org=all-island-research-observatory&org=an-garda-siochana&org=benefacts&org=bord-iascaigh-mhara&org=central-applications-office&org=central-bank-of-ireland&org=central-statistics-office&org=coca-cola-zero-bikes&org=commissioners-of-irish-lights&org=commission-for-communications-regulation&org=commission-for-energy-regulation&org=companies-registration-office&org=cork-city-council&org=department-of-agriculture-food-and-the-marine&org=department-of-jobs-enterprise-and-innovation&org=department-of-children-and-youth-affairs&org=department-of-communications-climate-action-and-environment&org=department-of-arts-heritage-regional-rural-and-gaeltacht-affairs&org=department-of-education-and-skills&org=department-of-finance&org=department-of-foreign-affairs-and-trade&org=department-of-health&org=department-of-housing-planning-and-local-government&org=department-of-justice-equality&org=department-of-public-expenditure-and-reform&org=department-of-social-protection&org=department-of-transport-tourism-and-sport&org=donegal-county-council&org=dublin-city-business-improvement-district&org=dublin-city-council&org=dublinked&org=dublinkedie&org=dun-laoghaire-rathdown-county-council&org=ehealth-ireland&org=eirgrid&org=environmental-protection-agency&org=european-commission&org=eurostat&org=failte-ireland&org=fingal-county-council&org=fixyourstreetie&org=food-safety-authority-of-ireland&org=forfas&org=galway-city-council&org=galway-county-council&org=geological-survey-of-ireland&org=health-and-safety-authority&org=health-insurance-authority&org=health-products-regulatory-authority&org=health-protection-surveillance-centre&org=health-research-board&org=health-service-executive&org=heritage-council&org=higher-education-authority&org=oireachtas&org=injuriesboardie&org=irish-maritime-development-office&org=irish-naturalisation-and-immigration-service&org=irish-naval-service&org=irish-prison-service&org=irish-sports-council&org=isde&org=kildare-county-council&org=legal-aid-board&org=lgma&org=limerick-city-county-council&org=marine-institute&org=marine-renewable-energy-ireland-marei&org=mayo-county-council&org=meteireann&org=money-advice-and-budgeting-service&org=national-biodiversity-data-centre&org=national-educational-welfare-board&org=national-transport-authority&org=nui-galway&org=nui-maynooth&org=oecd&org=office-of-public-works&org=office-of-the-attorney-general&org=office-of-the-director-of-public-prosecutions&org=office-of-the-refugee-applications-commissioner&org=ordnance-survey-ireland&org=pharmaceutical-society-of-ireland&org=private-residential-tenancies-board&org=property-services-regulatory-authority&org=quality-and-qualifications-ireland&org=railway-procurement-agency&org=reception-and-integration-agency&org=revenue-commissioners&org=road-safety-authority&org=roscommon-county-council&org=royal-irish-academy&org=science-foundation-ireland&org=sea-fisheries-protection-authority&org=smartbay-ireland&org=solas&org=south-dublin-county-council&org=sport-ireland&org=statcentral&org=state-examinations-commission&org=sustainable-energy-authority-of-ireland&org=teagasc&org=the-adoption-authority-of-ireland&org=the-courts-service&org=the-national-cancer-registry&org=the-national-cancer-screening-service&org=the-national-treatment-purchase-fund&org=the-probation-service&org=transport-infrastructure-ireland&org=university-college-dublin&org=valuation-office&org=w3c_consortium&org=westmeath-county-council&org=wexford-county-council&licence=CC0-1.0&licence=CC-BY-4.0&licence=CC-BY-SA-4.0&licence=cc-by-sa&licence=cc-by-nc-nd&licence=GFDL-1.3-no-cover-texts-no-invariant-sections&licence=psi&licence=notspecified&licence=me-custom&licence=oir-psi&licence=ODC-BY-1.0&licence=ODbL-1.0&licence=ODC-PDDL-1.0&licence=OGL-UK-2.0&licence=other-at&licence=other-nc&licence=other-closed&licence=other-open&licence=other&licence=other-pd&theme=Agriculture&theme=Crime&theme=Economy&theme=Education+and+Sport&theme=Energy&theme=Environment&theme=Government&theme=Health&theme=Housing&theme=Society&theme=Science&theme=Towns&theme=Transport&format=TIFF&created_after=&created_before=&modified_after=&modified_before= https://data.gov.ie/advanced_search?query=&texttype=anywhere...
- eruci 8y ago
  Are you going to release any street address datasets at some point?
  - wiredfool 8y ago
    I believe that the eircode dataset is one of the most highly requested sources, but it’s a private for profit database. If you are aware of such a dataset that a public body is hosting, then it would certainly be something to include. Convincing (and helping) the public bodies to publish their data is still a big task.
    - disgruntledphd2 8y ago
      I believe that An Post built a similar system many moons ago, and they should presumably be more amenable to open-sourcing. The handling of the whole Eircode thing makes my blood boil, to be honest.
- fnord123 8y ago
  Playing around, a lot of countries use this url scheme: https://data.gov.be https://data.gov.be http://data.gov.ro/ http://data.gov.ro/ https://www.dati.gov.it/ https://www.dati.gov.it/ (note, Italy redirects data. to dati.) https://data.gov.pl/ https://data.gov.pl/ https://data.gov.za/ https://data.gov.za/ (South Africa's has a cert problem) https://data.gov.in https://data.gov.in https://data.gov.au/ https://data.gov.au/ https://datos.gob.mx/ https://datos.gob.mx/
  - snaily 8y ago
    https://data.gov.sg/ https://data.gov.sg/ (Singapore)
  - vquemener 8y ago
    https://www.data.gouv.fr https://www.data.gouv.fr (France)
  - nonnon 8y ago
    https://www.data.gv.at/ https://www.data.gv.at/ (Austria)
  - mnx 8y ago
    although data.gov.pl exists, and even presents a valid cert, it has no content. The place with the data is: https://danepubliczne.gov.pl/en/# https://danepubliczne.gov.pl/en/# (website is being terribly slow for me right now)
  - Moghammed 8y ago
    https://data.overheid.nl/ https://data.overheid.nl/
  - divbyzer0 8y ago
    https://data.gov.ie/ https://data.gov.ie/
alphakappa 8y ago
Also, https://data.gov.in https://data.gov.in -- Lots of self-submitted datasets, and includes an API. Usability is average, much like the other government data sources.
philipashlock 8y ago
@danso thanks for the feedback on data.gov. I'm part of the small (3 person) team that helps to manage it. If you have a moment to chat I'll see if I can reach out to you to see if you'd be interested in participating in some more in-depth user research in the future. Folks can also always leave feedback via email, github, twitter, and other means - https://www.data.gov/contact https://www.data.gov/contact The Federal Data Strategy will also be opening up for comments again in October - https://strategy.data.gov/feedback/ https://strategy.data.gov/feedback/ Data.gov and Federal agencies use the same metadata standard (DCAT) that Google Dataset Search is using so much of our metadata is also being syndicated there.
- thebooglebooski 8y ago
  I think the biggest blocker with using publicly available datasets is stale data. If you, or anyone else who aggregates these datasets could make it EASY to find the FREQUENCY of updates, rather than just the LAST UPDATED timestamp, it'd incentivize people to consume APIs more. I realize having a snapshot from 2014 is better than what was publicly available before. But I feel no one's really talked about why they would or wouldn't use particular data.
  - izyda 8y ago
    I think this is exactly correct. Frequency of updates (and clear documentation of the lag relationship between when data is reported and for what period data is applicable too) is often missing or hard to find. The value of increasing the cadence of updates should also not be understated! A lot of public dataset report on annual frequencies with more than a quarter of delay... Although this is a different issue altogether that has more to do with the processes of the reporting agency.
    - chetanbhasin 8y ago
      Yes, it's interesting how much difference the data about data management can make in people's engagement with the platform.
- 8y ago
Cd00d 8y ago
I really like NYC Open Data: https://opendata.cityofnewyork.us/ https://opendata.cityofnewyork.us/
Untit1ed 8y ago
Blatantly self-promoting, but we (in Australia) are trying to develop a general solution for better open data search over at https://search.data.gov.au https://search.data.gov.au - solution is all open source at https://github.com/magda-io/magda https://github.com/magda-io/magda.
rufuspollock 8y ago
Interesting. There have been a lot of attempts at "meta data portals" that search across portals. Most of them have struggled. At Open Knowledge we built a really early one called opendatasearch.org in 2011/2012 - now defunct - and were involved in the first version of the pan EU open data portal. We also had the original https://ckan.net/ https://ckan.net/ (and subsites) which is now https://datahub.io/ https://datahub.io/ and has become much more focused on quality data and data deployment. [Disclosure: I was/am involved in many of these projects] The challenge, as others have mentioned, is that data quality is very variable and searching for datasets is complicated (think of software as an analogy - searching for good code libraries is a bit of an art). I imagine Google are trying this out before making datasets another "special type" of search result -- after all you can already search google for datasets. In addition, Google are already Google so including datasets will have a level of comprehensiveness and exposure you struggle with elsewhere (part of the power of monopoly in a sense!). PS: for those looking for data gov sites https://dataportals.org/ https://dataportals.org/ has most of these.

minimaxir 8y ago

Interesting vertical integration for Kaggle Datasets: https://www.kaggle.com/datasets https://www.kaggle.com/datasets

rufuspollock 8y ago
Note Kaggle recently adopted the FrictionlessData.io Data Package specs: https://github.com/Kaggle/kaggle-api/wiki/Dataset-Metadata https://github.com/Kaggle/kaggle-api/wiki/Dataset-Metadata

afettere 8y ago

https://data.urbanfootprint.com/ https://data.urbanfootprint.com/ - browse thousands of environmental, social, transportation, and land use datasets. (disclosure - I work there)

Dangeranger 8y ago

Nice. I'm glad Google is making it easier to find public data sets. I wish that these could be filtered by format, so that you could narrow them to CSV, XML, JSON, KML, etc. Another nice resource that I've used in the past is 'toddmotto/public-apis' on Github [0]. In the end I would prefer all public data sets to be available over the DAT protocol [1] instead of being hosted only on government or organization websites. A lot of climate data previously made available by the EPA was taken down, and only saved by efforts of volunteers.[2] [0] https://github.com/toddmotto/public-apis https://github.com/toddmotto/public-apis [1] https://datproject.org/ https://datproject.org/ [2] https://www.wired.com/2017/01/rogue-scientists-race-save-climate-data-trump/ https://www.wired.com/2017/01/rogue-scientists-race-save-cli...

neuromantik8086 8y ago
Dat's pretty cool, but it's not the only game out there. The efforts of git-annex/Datalad [0], Academic Torrents [1], Quilt[2], DVC[3], and Pachyderm [4] are also notable in this space. My hopes are broader in the sense that I just hope that dataset versioning happens in the first place. [0] https://www.datalad.org/ https://www.datalad.org/ [1] http://academictorrents.com/ http://academictorrents.com/ [2] https://quiltdata.com/ https://quiltdata.com/ [3] https://dvc.org/ https://dvc.org/ [4] http://www.pachyderm.io/ http://www.pachyderm.io/
- rgardaphe 8y ago
  [5] https://qri.io https://qri.io (we're tackling the dataset versioning problem head on.
sixdimensional 8y ago
Yeah, if they had added an attribute for formats available for each data set, and then added a filter by type to the search (e.g. "type:csv") or similar, it would be great. Sometimes you really want a specific format for a dataset.
- matt4077 8y ago

ericand 8y ago

What's up with this domain? Is there anything else under toolbox.google.com?

deleted 8y ago
[deleted]
amadeuspzs 8y ago
Here you go: http://bfy.tw/JkG7 http://bfy.tw/JkG7
- dang 8y ago
  Please don't do this here.

jonbaer 8y ago

Using Google to search Google ... https://toolbox.google.com/datasetsearch/search?query=site%3Agoogle.com https://toolbox.google.com/datasetsearch/search?query=site%3...

neuromantik8086 8y ago

Looks like academic institutional repositories and figshare are doing the heavy lifting here. It's still neat to see Google aggregate everything, but it's not that different from what they do with other services relying on these sources already, and is largely dependent upon how rich these upstream sources are in the first place.

earth2mars 8y ago

It would be good to know who sponsored the data research so that everyone can make self biased decision to use/trust the data.

MrEfficiency 8y ago

What is the best way for a website to format data to the public? I already have my presentation, but I can also provide it as a .xls, .csv, sql, or html table. What would be best to help programmers/data scientists use my data?

polkapolka 8y ago
Csv and a permitive license like CC0. Provide good meta data to make provenance easier. Mark up your dataset with schema.org.
combatentropy 8y ago
I vote CSV. - simple - lightweight - open It's easy for the consumer of your data to convert a CSV to whatever format they need. - spreadsheet, for personal analysis - SQL database, for industrial-strength analysis - HTML, for pretty output to their users
- lolive 8y ago
  Oh, and don't forget that CSV is a rigorous data format, with many tricks. Don't just append some text together, separated by commas, and call it CSV. Instead, use a dedicated lib to create the CSV for you.
heinrichhartman 8y ago
1) Put it on figshare or github. Your website might be dead next year. Those services have a better chance of survival. 2) CSV, JSON works fine if your dataset is just a few numbers and strings. GitHub will preview csv files as HTML tables. 3) If you need the efficiency of binary data and more robust data containers I would look into - Parquet https://parquet.apache.org/documentation/latest/ https://parquet.apache.org/documentation/latest/ and - Arvo http://avro.apache.org/ http://avro.apache.org/ http://blog.cloudera.com/blog/2009/11/avro-a-new-format-for-data-interchange/ http://blog.cloudera.com/blog/2009/11/avro-a-new-format-for-... 4) DataScientists work with R/Pandas "DataFrames". If you are familiar with either one, import the data into a data frame and use an export method to do the serialization for you: https://pandas.pydata.org/pandas-docs/stable/api.html#id12 https://pandas.pydata.org/pandas-docs/stable/api.html#id12

michaelmior 8y ago

Of note is the link below which indicates how you can have your dataset indexed. https://developers.google.com/search/docs/data-types/dataset https://developers.google.com/search/docs/data-types/dataset

xaranke 8y ago

Placeholder comment to calculate the time before Google inevitably shuts this down.

heinrichhartman 8y ago

The state of data sharing seems to be still quite sad. * Hosting problems. The first link I tried was already broken. * Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?). * Update problems. Many data-sets change over time (e.g. GDP). How can I subscribe to updates? "git pull" would be nice. * Provenance problems. I want to know who put which record into the dataset, when and why? "git log" would be nice. * Presentation problems. (This is OK sometimes) I necessarily want to download 5Gb file before I looked into it. The first few rows of the dataset should be presented on the page, with information about it. Wrote down a few more thoughts a while ago here: https://github.com/HeinrichHartmann/data-sharing#in-the-ideal-world https://github.com/HeinrichHartmann/data-sharing#in-the-idea... Approaches I have seen so far in the wild: * figshare.com -- Addresses Hosting and Presentation. * https://quiltdata.com/ https://quiltdata.com/ -- (!) looks great. Still exploring. * github.com -- works fine for small datasets (<1GB) * packaging (yum, pkg, pip) -- (?) Not sure if that works, but at least they solve: Hosting, Update, Provenance. This seems to be a wide open problem to me.

rgardaphe 8y ago
Totally agree! At Qri (https://qri.io https://qri.io) we're working on many of these problems together - hosting, formatting (interoperability), provenance and sync. It's an open source project - we'd love to have your feedback as we design it!
- sixdimensional 8y ago
  This is cool and is a perfect case for IPFS for public datasets. I've not heard of it before, though, and I think naming / branding is something that makes finding these things / building momentum more difficult. For example, someone else mentioned engima.com. I would have no idea that is related to data sources / sets unless I knew what it was. Certainly wish you the best of luck though and will keep an eye on Qri! Cool project!

rebel1 8y ago

Interestingly enough, I built and released something very similar [1] about a month ago using a Google Custom Search Engine. Here is the Show HN for it: https://news.ycombinator.com/item?id=17789119 https://news.ycombinator.com/item?id=17789119 [1] https://databasd.com/ https://databasd.com/

spoiledtechie 8y ago

Can we list some good data searches to suggest below this comment? I would love to see some cool data we might be able to use.

sbr464 8y ago

This is nice. I'm working on a similar open source project that is releasing soon called DataLibrary[0] It goes further by bringing this kind of data together into a single API, converting/cleaning into a similar schema where possible. A small write up can be found on github [1]. Any feedback/ideas would be appreciated! [0] https://www.datalibrary.com https://www.datalibrary.com (not online currently) [1] https://github.com/reactual/datalibrary/blob/master/README.md https://github.com/reactual/datalibrary/blob/master/README.m...

madisonmay 8y ago

I also very much like https://www.figure-eight.com/data-for-everyone/ https://www.figure-eight.com/data-for-everyone/. It's not optimized for search but it's an excellent repository of high quality datasets.

smartvlad 8y ago

Look at https://knoema.com https://knoema.com which positions itself as a search engine for data with more than 2.5 billion time series available. They provide both visual data discovery through search and navigation as well as API access through Python, R etc.

ToFab123 8y ago

Pretty lame that it doesn't work in Edge.

choicenotchance 8y ago

Most of them are paid.

sytelus 8y ago

Looks like this doesn’t crawl one of the most important source for data: academic torrents. For example, I searched for ilsvrc lmdb This should have found imagenet data in lmdb format available somewhere but it returned no results.

frabcus 8y ago

Nice to see Google trying this again! It's one of those areas they have long attempts at involvement in - e.g. Google Public Data Explorer which never quite reached it's potential, and Freebase which although flawed was good and was shut down after Google acquired it. I like that this is search based! The web is still the best place to publish data - in fact in my view normal Google search is still by far the best way to find datasets, even though it isn't directly designed for that. There's a link from the about page of Google Dataset Search to this help for webmasters on how to mark up content for it - although it is a bit odd, mainly showing how to mark a dataset with a DOI (so good for academics certainly!): https://productforums.google.com/forum/#!topic/webmasters/nPq4BW6iPIA https://productforums.google.com/forum/#!topic/webmasters/nP... Just metadata about data feels like a very niche thing to search to me - I'm still not convinced anyone will maintain the metadata well enough to help. Possibly will work in particular domains. Does Dataset Search have some way to search column headings, types or content (of CSV, Excel, JSON etc)? I can imagine a load of operators that would make that really powerful for finding badly meta-marked up datasets deep in the web. Would seem like the obvious extra thing a dataset search would do. Also previews please!!! Just nicely render the fist ten rows of common formats - CSV and Excel to begin with. What part of Google is doing this?

a2x 8y ago

Google is so damn bad on UX. An excellent example of interaction of unlimited ressources and priorities.

nterpo 8y ago

You may want to try [https://data.opendatasoft.com](https://data.opendatasoft.com) https://data.opendatasoft.com](https://data.opendatasoft.com... -- thousands of datasets available through the same API, usable online, no download required.

xcubic 8y ago

Add this one: https://dados.gov.pt/ https://dados.gov.pt/ (Portugal)

deleted 8y ago

[deleted]

jwillmer 8y ago

Since we have already a nice reference list of open data portals by country I like to add the German portal: https://www.govdata.de/ https://www.govdata.de/

deleted 8y ago

[deleted]

lmy86263 8y ago

it will be helpful for somebody who major in AI

sprague 8y ago

Then there's this $500K just awarded by the NSF to build a "Google for data sets". I wonder if, before making these sorts of grants, the NSF looks at what Google and other and other companies are already (or likely) doing. https://www.lehigh.edu/engineering/news/faculty/2018/20180820-davison-heflin-jia-dataset-search-engine-nsf-award.html https://www.lehigh.edu/engineering/news/faculty/2018/2018082...

MR4D 8y ago

Maybe I’m missing something but this strikes me as underwhelming. To the point of something that I could do as opposed to something that the firm that created maps and gmail could do. Is it just me?

ece 8y ago

Nothing for FRED https://toolbox.google.com/datasetsearch/search?query=site%3Afred.stlouisfed.org https://toolbox.google.com/datasetsearch/search?query=site%3... hmm...