12 ms·

Google Dataset Search

388 points by abraxaz 5y ago

abraxaz 5y ago
Information on how to annotate datasets: https://developers.google.com/search/docs/data-types/dataset https://developers.google.com/search/docs/data-types/dataset > We can understand structured data in Web pages about datasets, using either schema.org Dataset markup, or equivalent structures represented in W3C's Data Catalog Vocabulary (DCAT) format. We also are exploring experimental support for structured data based on W3C CSVW, and expect to evolve and adapt our approach as best practices for dataset description emerge. For more information about our approach to dataset discovery, see Making it easier to discover datasets. For more info on those: - W3C's Data Catalog Vocabulary: https://www.w3.org/TR/vocab-dcat-3/ https://www.w3.org/TR/vocab-dcat-3/ - Schema.org dataset: https://schema.org/Dataset https://schema.org/Dataset - CSVW Namespace Vocabulary Terms: https://www.w3.org/ns/csvw https://www.w3.org/ns/csvw - Generating RDF from Tabular Data on the Web (examples on how to use CSVW): https://www.w3.org/TR/csv2rdf/ https://www.w3.org/TR/csv2rdf/
- westurner 5y ago
  Use cases for such [LD: Linked Data] metadata: 1. #StructuredPremises: > (How do I indicate that this is a https://schema.org/ScholarlyArticle https://schema.org/ScholarlyArticle predicated upon premises including this Dataset and these logical propositions?) 2. #LinkedMetaAnalyses; #LinkedResearch "#StudyGraph" 3. [CSVW (Tabular Data Model),] schema.org/Dataset(s) with per column (per-feature) physical quantity and unit URIs with e.g. QUDT and/or https://schema.org/StructuredValue https://schema.org/StructuredValue metadata for maximum data reusability. 4. JupyterLab notebooks: 4a. JupyterLab Metadata Service extension: https://github.com/jupyterlab/jupyterlab-metadata-service https://github.com/jupyterlab/jupyterlab-metadata-service : > - displays linked data about the resources you are interacting with in JuyterLab. > - enables other extensions to register as linked data providers to expose JSON LD about an entity given the entity's URL. > - exposes linked data to the user as a Linked Data viewer in the Data Browser pane. 4b. JupyterLab Data Explorer: https://github.com/jupyterlab/jupyterlab-data-explorer https://github.com/jupyterlab/jupyterlab-data-explorer : > - Data changing on you? Use RxJS observables to represent data over time. > - Have a new way to look at your data? Create React or lumino components to view a certain type. > - Built-in data explorer UI to find and use available datasets.
- prepend 5y ago
  It’s funny because Google does not use these standards to validate. I keep getting errors from Google that some of my dataset’s descriptions are over 5,000 characters even though dcat:description does not have a size limit. Of course it’s impossible for me to report a bug in how they index.
  - hans1729 5y ago
    You could submit a dataset containing the bug report :-)
davcancas 5y ago
This dataset search engine has been around for years! We created DataMarket (https://datamarket.es https://datamarket.es) inspired by this site (and Auren Hoffman's SafeGraph).
- vkhuc 5y ago
  Absolutely. This dataset search was first introduced in September of 2018. It was out of beta in January last year: https://www.kdnuggets.com/2020/01/google-dataset-search.html https://www.kdnuggets.com/2020/01/google-dataset-search.html.
fabcomm 5y ago
This is every data scientists’ dream.
- villasv 5y ago
  This is not new, though. So it may be a dream in the sense of people have been asleep?
  - sneilan1 5y ago
    How long until Google shuts this service down?
  - puzzledobserver 5y ago
    Why the condescension? Do you mean that Google has been offering this service for a while? Or do you mean that similar services have previously been offered by other organizations? In which case, perhaps you could link to them?
    - villasv 5y ago
      No condescension intended. Google has been offering this service for a while, at least since 2018 iirc.
- prepend 5y ago
  It’s ok, but surprisingly feature poor since they only index datasets with structured metadata. I kind of wish they would compile all their metadata into a structured mega-catalog and allowed searching by api. Or just dumped it out as a dataset itself.
  - chatmasta 5y ago
    Then you'll love what we're doing at Splitgraph: https://www.splitgraph.com/connect https://www.splitgraph.com/connect As far as your SQL client is concerned, data.splitgraph.com:5432 is a giant Postgres database with ~40,000 tables in it. You can query and join across them using your existing tools. Behind the curtain, we'll forward your query to the upstream data source, translating it from SQL to whatever language it expects. (We can also ingest delta-compressed versioned snapshots).
Der_Einzige 5y ago
Stop, you're making the barrier to entry too low! /s This is really really cool. Between this and Hugginfaces Dataset and models hubs, AI/ML is really getting easier to use.
- paulz_ 5y ago
  I've actually been on the lookout for model hubs lately. Any that you've seen or reccomend? I've found https://modelzoo.co/ https://modelzoo.co/ but it seems more like a currated list of models (some incomplete) rather than a community where users share trained models.
  - halflings 5y ago
    If you use tensorflow, tfhub [1] is the go-to model repository. [1] https://tfhub.dev/ https://tfhub.dev/
uptime 5y ago
I have a lot to read before I get excited but if the team is here: Can we get DCAT for sets that are otherwise only discoverable with OAI-PMH? Seems like a divide between govt and academic repos that hinders harvesting.
sigmonsays 5y ago
Do they have a deprecation notice up already?
- vallas 5y ago
  Everyone's hyped but Vanilla Google is even better in term of design: Google.com is more readable, just like Hackernews.
a_square_peg 5y ago
Also in case the team is here... the updated date for ERA5 back extension to 1950-1978 (Preliminary version - https://datasetsearch.research.google.com/search?query=ERA5%20atmospheric%20reanalysis&docid=L2cvMTFwM2tiM3NmYw%3D%3D https://datasetsearch.research.google.com/search?query=ERA5%...) is incorrect as this was only released last year (2020) but is stated as 2011.
chatmasta 5y ago
This is a great resource. At Splitgraph, we index ~40k open data sets, and we make sure to include structured metadata for each one, so we show up in these results. (example [0]) One cool aspect of this metadata is that it allows a dataset to have multiple sources. So if two sites index the same dataset, there is no duplicate content penalty like there might be with textual content. If you search for a dataset, it will include links to all its sources (whether canonical or otherwise). For most of the data we index at Splitgraph, the canonical source is an open government data portal powered by Socrata (e.g. data.cdc.gov). We noticed that Socrata powered a lot of portals, so we wrote a Socrata plugin for Splitgraph, along with a scraper to index the metadata. The plugin basically implements a Postgres FDW so that Splitgraph can translate from SQL to the upstream query language. In this case, the plugin translates to Socrata's bespoke API language. But for private deployments we also have plugins for Snowflake, Postgres, some SaaS services, etc. If you find some data on Google Dataset Search with Splitgraph listed as a source, please take a look! Our "Data Delivery Network" (DDN) is implemented on top of the Postgres wire protocol, so you can connect with any Postgres client (or use our web editor). All the Postgres query syntax is available to you; you can even JOIN across any of the other 40k+ datasets indexed at Splitgraph. That includes "live data" like Socrata portals, but also versioned snapshots of data called "data images." Here's an example of a point-in-time query across two snapshots (basically a diff) [1], and another query that joins across tables at data.cityofchicago.org and data.cambridgema.gov [2]. [0] https://www.splitgraph.com/cdc-gov/distribution-of-covid19-deaths-and-populations-by-jwta-jxbg https://www.splitgraph.com/cdc-gov/distribution-of-covid19-d... – "View Source" to see the Schema.org metadata [1] https://bit.ly/3epvxcj https://bit.ly/3epvxcj [2] https://bit.ly/3f1ll8K https://bit.ly/3f1ll8K (Sorry for the bit.ly links. The URL for our query editor includes the full SQL string, and I don't want to mess up HN formatting.)
- remram 5y ago
  Why did you make your website's scrollbar half the normal size? It looks the same as native, except too small to grab? Why?!
john-tells-all 5y ago
Dataset with 9,000 annotated cat images! => https://datasetsearch.research.google.com/search?query=cat&docid=L2cvMTFqY2tkNTI3MQ%3D%3D https://datasetsearch.research.google.com/search?query=cat&d...
paulcole 5y ago
I was looking up dentistry data sets (my industry) and came across this: https://www.arcgis.com/home/item.html?id=9850793c688e4eebaab8be8e98cc6b28 https://www.arcgis.com/home/item.html?id=9850793c688e4eebaab... Can anybody explain why this showed up in a dataset search and what exactly the data is?
- ladberg 5y ago
  Someone trying to do SEO by getting links to their site from a popular one?
  - runj__ 5y ago
    The person who added the data set is the same person that owns the business, so this seems pretty likely.
lunatuna 5y ago
I'm not a scientist, so not the most scholarly first lookup, but tried searching for penis data[0]. The first link sent me to a site that requires signup to use [1]. No fun. Won't use again. [0] - https://datasetsearch.research.google.com/search?query=penis&docid=L2cvMTFqbl82NWZ3Ng%3D%3D https://datasetsearch.research.google.com/search?query=penis... [1] - https://data.world/jemus42/world-penis-data https://data.world/jemus42/world-penis-data
- alphachloride 5y ago
  I'll keep using it because the inconvenience of the occasional sign-up is trumped by the convenience of searchable datasets.
smhx 5y ago
another good resource that's more specific to machine learning is https://paperswithcode.com/datasets https://paperswithcode.com/datasets
plaidfuji 5y ago
I’ve come across this a few different times over the years... always seems enticing and potentially useful, but I’ve never found a real use for it. I suppose it provides a library of well-prepped datasets to test ML models on? Anyone ever used this for any practical purpose beyond a sandbox-type use case?
- disgruntledphd2 5y ago
  Well, you could use it to answer questions of interest to you. I certainly tried my current projects on it, and found some useful stuff (most of which I've seen before).
apprenticer 5y ago
The privacy problem should be considered
- drusepth 5y ago
  Can you explain what you mean by "the privacy problem"?
- mrkramer 5y ago
  Privacy is the liability of those who collect data and share it in a form of dataset not of those who search for datasets or who crawl them.
igravious 5y ago
Discussion from Jan 2020: https://news.ycombinator.com/item?id=22130874 https://news.ycombinator.com/item?id=22130874 | 32 comments Discussion from Sept 2018: https://news.ycombinator.com/item?id=17919297 https://news.ycombinator.com/item?id=17919297 | 76 comments
jeresuikkila 5y ago
They should add a "I'm Feeling lucky" button
ravila4 5y ago
The lab I work in has a project that helps annotate datasets with metadata and register their schemas: https://discovery.biothings.io/ https://discovery.biothings.io/ A common barrier to making FAIR datasets is that not all data lends itself to be schema.org compliant. The idea is that instead of enforcing one schema to rule them all, we allow people to make their own schemas by extending existing ones, and register them in an API to be easily discoverable.
damirkotoric 5y ago
Shameless plug. I wrote a piece about The State of Open Data Portals https://uxdesign.cc/designing-open-data-portals-for-government-85e2524f5877 https://uxdesign.cc/designing-open-data-portals-for-governme... where I predict that it'll take a Google to really provide a single searchable dataset portal for the whole world. Doesn't take a genius to predict, but there ya go! Governments are assembling datasets in a very fragmented way. It'll take a private company to provide one single website to explore and find all datasets from around the world, making it easier to look at holistic patterns that are happening around the world, or compare patterns between countries. Though, I would expect a much better UX from Google nowadays. This site has more in common with Google Scholar than Google Search. And ultimately I'd like to see them build something where people don't need to download datasets in order to make use of the data. I compare the state of open data to the state of mapping software before Google Maps. You needed to download map files and open them on special software that you open on your computer to make sense of the data. And then Google Maps came along and flipped that whole model. Open data needs the same leap forward in order for more people to make greater use of open data.
antpls 5y ago
No mention of https://frictionlessdata.io/ https://frictionlessdata.io/ dataset metadata format which is also used by Kaggle

388 points by abraxaz 5y ago

Information on how to annotate datasets: https://developers.google.com/search/docs/data-types/dataset https://developers.google.com/search/docs/data-types/dataset > We can understand structured data in Web pages about datasets, using either schema.org Dataset markup, or equivalent structures represented in W3C's Data Catalog Vocabulary (DCAT) format. We also are exploring experimental support for structured data based on W3C CSVW, and expect to evolve and adapt our approach as best practices for dataset description emerge. For more information about our approach to dataset discovery, see Making it easier to discover datasets. For more info on those: - W3C's Data Catalog Vocabulary: https://www.w3.org/TR/vocab-dcat-3/ https://www.w3.org/TR/vocab-dcat-3/ - Schema.org dataset: https://schema.org/Dataset https://schema.org/Dataset - CSVW Namespace Vocabulary Terms: https://www.w3.org/ns/csvw https://www.w3.org/ns/csvw - Generating RDF from Tabular Data on the Web (examples on how to use CSVW): https://www.w3.org/TR/csv2rdf/ https://www.w3.org/TR/csv2rdf/

westurner 5y ago
Use cases for such [LD: Linked Data] metadata: 1. #StructuredPremises: > (How do I indicate that this is a https://schema.org/ScholarlyArticle https://schema.org/ScholarlyArticle predicated upon premises including this Dataset and these logical propositions?) 2. #LinkedMetaAnalyses; #LinkedResearch "#StudyGraph" 3. [CSVW (Tabular Data Model),] schema.org/Dataset(s) with per column (per-feature) physical quantity and unit URIs with e.g. QUDT and/or https://schema.org/StructuredValue https://schema.org/StructuredValue metadata for maximum data reusability. 4. JupyterLab notebooks: 4a. JupyterLab Metadata Service extension: https://github.com/jupyterlab/jupyterlab-metadata-service https://github.com/jupyterlab/jupyterlab-metadata-service : > - displays linked data about the resources you are interacting with in JuyterLab. > - enables other extensions to register as linked data providers to expose JSON LD about an entity given the entity's URL. > - exposes linked data to the user as a Linked Data viewer in the Data Browser pane. 4b. JupyterLab Data Explorer: https://github.com/jupyterlab/jupyterlab-data-explorer https://github.com/jupyterlab/jupyterlab-data-explorer : > - Data changing on you? Use RxJS observables to represent data over time. > - Have a new way to look at your data? Create React or lumino components to view a certain type. > - Built-in data explorer UI to find and use available datasets.
prepend 5y ago
It’s funny because Google does not use these standards to validate. I keep getting errors from Google that some of my dataset’s descriptions are over 5,000 characters even though dcat:description does not have a size limit. Of course it’s impossible for me to report a bug in how they index.
- hans1729 5y ago
  You could submit a dataset containing the bug report :-)

davcancas 5y ago

This dataset search engine has been around for years! We created DataMarket (https://datamarket.es https://datamarket.es) inspired by this site (and Auren Hoffman's SafeGraph).

vkhuc 5y ago
Absolutely. This dataset search was first introduced in September of 2018. It was out of beta in January last year: https://www.kdnuggets.com/2020/01/google-dataset-search.html https://www.kdnuggets.com/2020/01/google-dataset-search.html.

fabcomm 5y ago

This is every data scientists’ dream.

villasv 5y ago
This is not new, though. So it may be a dream in the sense of people have been asleep?
- sneilan1 5y ago
  How long until Google shuts this service down?
- puzzledobserver 5y ago
  Why the condescension? Do you mean that Google has been offering this service for a while? Or do you mean that similar services have previously been offered by other organizations? In which case, perhaps you could link to them?
  - villasv 5y ago
    No condescension intended. Google has been offering this service for a while, at least since 2018 iirc.
prepend 5y ago
It’s ok, but surprisingly feature poor since they only index datasets with structured metadata. I kind of wish they would compile all their metadata into a structured mega-catalog and allowed searching by api. Or just dumped it out as a dataset itself.
- chatmasta 5y ago
  Then you'll love what we're doing at Splitgraph: https://www.splitgraph.com/connect https://www.splitgraph.com/connect As far as your SQL client is concerned, data.splitgraph.com:5432 is a giant Postgres database with ~40,000 tables in it. You can query and join across them using your existing tools. Behind the curtain, we'll forward your query to the upstream data source, translating it from SQL to whatever language it expects. (We can also ingest delta-compressed versioned snapshots).

Der_Einzige 5y ago

Stop, you're making the barrier to entry too low! /s This is really really cool. Between this and Hugginfaces Dataset and models hubs, AI/ML is really getting easier to use.

paulz_ 5y ago
I've actually been on the lookout for model hubs lately. Any that you've seen or reccomend? I've found https://modelzoo.co/ https://modelzoo.co/ but it seems more like a currated list of models (some incomplete) rather than a community where users share trained models.
- halflings 5y ago
  If you use tensorflow, tfhub [1] is the go-to model repository. [1] https://tfhub.dev/ https://tfhub.dev/

uptime 5y ago

I have a lot to read before I get excited but if the team is here: Can we get DCAT for sets that are otherwise only discoverable with OAI-PMH? Seems like a divide between govt and academic repos that hinders harvesting.

sigmonsays 5y ago

Do they have a deprecation notice up already?

vallas 5y ago
Everyone's hyped but Vanilla Google is even better in term of design: Google.com is more readable, just like Hackernews.

a_square_peg 5y ago

Also in case the team is here... the updated date for ERA5 back extension to 1950-1978 (Preliminary version - https://datasetsearch.research.google.com/search?query=ERA5%20atmospheric%20reanalysis&docid=L2cvMTFwM2tiM3NmYw%3D%3D https://datasetsearch.research.google.com/search?query=ERA5%...) is incorrect as this was only released last year (2020) but is stated as 2011.

chatmasta 5y ago

This is a great resource. At Splitgraph, we index ~40k open data sets, and we make sure to include structured metadata for each one, so we show up in these results. (example [0]) One cool aspect of this metadata is that it allows a dataset to have multiple sources. So if two sites index the same dataset, there is no duplicate content penalty like there might be with textual content. If you search for a dataset, it will include links to all its sources (whether canonical or otherwise). For most of the data we index at Splitgraph, the canonical source is an open government data portal powered by Socrata (e.g. data.cdc.gov). We noticed that Socrata powered a lot of portals, so we wrote a Socrata plugin for Splitgraph, along with a scraper to index the metadata. The plugin basically implements a Postgres FDW so that Splitgraph can translate from SQL to the upstream query language. In this case, the plugin translates to Socrata's bespoke API language. But for private deployments we also have plugins for Snowflake, Postgres, some SaaS services, etc. If you find some data on Google Dataset Search with Splitgraph listed as a source, please take a look! Our "Data Delivery Network" (DDN) is implemented on top of the Postgres wire protocol, so you can connect with any Postgres client (or use our web editor). All the Postgres query syntax is available to you; you can even JOIN across any of the other 40k+ datasets indexed at Splitgraph. That includes "live data" like Socrata portals, but also versioned snapshots of data called "data images." Here's an example of a point-in-time query across two snapshots (basically a diff) [1], and another query that joins across tables at data.cityofchicago.org and data.cambridgema.gov [2]. [0] https://www.splitgraph.com/cdc-gov/distribution-of-covid19-deaths-and-populations-by-jwta-jxbg https://www.splitgraph.com/cdc-gov/distribution-of-covid19-d... – "View Source" to see the Schema.org metadata [1] https://bit.ly/3epvxcj https://bit.ly/3epvxcj [2] https://bit.ly/3f1ll8K https://bit.ly/3f1ll8K (Sorry for the bit.ly links. The URL for our query editor includes the full SQL string, and I don't want to mess up HN formatting.)

remram 5y ago
Why did you make your website's scrollbar half the normal size? It looks the same as native, except too small to grab? Why?!

john-tells-all 5y ago

Dataset with 9,000 annotated cat images! => https://datasetsearch.research.google.com/search?query=cat&docid=L2cvMTFqY2tkNTI3MQ%3D%3D https://datasetsearch.research.google.com/search?query=cat&d...

paulcole 5y ago

I was looking up dentistry data sets (my industry) and came across this: https://www.arcgis.com/home/item.html?id=9850793c688e4eebaab8be8e98cc6b28 https://www.arcgis.com/home/item.html?id=9850793c688e4eebaab... Can anybody explain why this showed up in a dataset search and what exactly the data is?

ladberg 5y ago
Someone trying to do SEO by getting links to their site from a popular one?
- runj__ 5y ago
  The person who added the data set is the same person that owns the business, so this seems pretty likely.

lunatuna 5y ago

I'm not a scientist, so not the most scholarly first lookup, but tried searching for penis data[0]. The first link sent me to a site that requires signup to use [1]. No fun. Won't use again. [0] - https://datasetsearch.research.google.com/search?query=penis&docid=L2cvMTFqbl82NWZ3Ng%3D%3D https://datasetsearch.research.google.com/search?query=penis... [1] - https://data.world/jemus42/world-penis-data https://data.world/jemus42/world-penis-data

alphachloride 5y ago
I'll keep using it because the inconvenience of the occasional sign-up is trumped by the convenience of searchable datasets.

smhx 5y ago

another good resource that's more specific to machine learning is https://paperswithcode.com/datasets https://paperswithcode.com/datasets

plaidfuji 5y ago

I’ve come across this a few different times over the years... always seems enticing and potentially useful, but I’ve never found a real use for it. I suppose it provides a library of well-prepped datasets to test ML models on? Anyone ever used this for any practical purpose beyond a sandbox-type use case?

disgruntledphd2 5y ago
Well, you could use it to answer questions of interest to you. I certainly tried my current projects on it, and found some useful stuff (most of which I've seen before).

apprenticer 5y ago

The privacy problem should be considered

drusepth 5y ago
Can you explain what you mean by "the privacy problem"?
mrkramer 5y ago
Privacy is the liability of those who collect data and share it in a form of dataset not of those who search for datasets or who crawl them.

igravious 5y ago

Discussion from Jan 2020: https://news.ycombinator.com/item?id=22130874 https://news.ycombinator.com/item?id=22130874 | 32 comments Discussion from Sept 2018: https://news.ycombinator.com/item?id=17919297 https://news.ycombinator.com/item?id=17919297 | 76 comments

jeresuikkila 5y ago

They should add a "I'm Feeling lucky" button

ravila4 5y ago

The lab I work in has a project that helps annotate datasets with metadata and register their schemas: https://discovery.biothings.io/ https://discovery.biothings.io/ A common barrier to making FAIR datasets is that not all data lends itself to be schema.org compliant. The idea is that instead of enforcing one schema to rule them all, we allow people to make their own schemas by extending existing ones, and register them in an API to be easily discoverable.

damirkotoric 5y ago

Shameless plug. I wrote a piece about The State of Open Data Portals https://uxdesign.cc/designing-open-data-portals-for-government-85e2524f5877 https://uxdesign.cc/designing-open-data-portals-for-governme... where I predict that it'll take a Google to really provide a single searchable dataset portal for the whole world. Doesn't take a genius to predict, but there ya go! Governments are assembling datasets in a very fragmented way. It'll take a private company to provide one single website to explore and find all datasets from around the world, making it easier to look at holistic patterns that are happening around the world, or compare patterns between countries. Though, I would expect a much better UX from Google nowadays. This site has more in common with Google Scholar than Google Search. And ultimately I'd like to see them build something where people don't need to download datasets in order to make use of the data. I compare the state of open data to the state of mapping software before Google Maps. You needed to download map files and open them on special software that you open on your computer to make sense of the data. And then Google Maps came along and flipped that whole model. Open data needs the same leap forward in order for more people to make greater use of open data.

antpls 5y ago

No mention of https://frictionlessdata.io/ https://frictionlessdata.io/ dataset metadata format which is also used by Kaggle