4 ms·
Is there a way to get a "dataset of datasets"? That is, all datasets you have, in downloadable tabular form with metadata for each dataset?
by rabidrat 7y ago
Is there a way to get a "dataset of datasets"? That is, all datasets you have, in downloadable tabular form with metadata for each dataset?
- kevinemoore 7y agoHmm, yes and no. It's certainly possible to nest a package (dataset) inside a package. But, there's no (current) API that returns the list of datasets as a table/DataFrame. How would you imagine using that feature?
- rabidrat 7y agoI use local tools for all my data processing, and if I want to search for a relevant dataset, I'd like to do it locally instead of having to contact an API and do all the paging etc. The kinds of queries that might be important to me are outside the scope of any fixed API; I might want all datasets updated in December (of any year), or only those with between 100k and 1m rows, or to get a frequency table of datasets by license terms. It's very easy to do any of these queries if I can download a TSV (or equivalent) with all the metadata, and usually too frustrating to even attempt if I can't.
- kevinemoore 7y agoMaybe the Elasticsearch cluster would help you here. Because all the datasets (and files within datasets) are indexed along with their metadata, you could write elastic queries to find the datasets you want--as long as the dataset creators are including the relevant metadata in the dataset annotations.
- akarve 7y agoThe package landing page is essentially that: a list of all of your datasets. This is constructed from the special s3://.quilt/* directory. Since a Quilt manifest is a list of keys, you can compose N manifests into a single manifest using the API, see e.g. https://docs.quiltdata.com/advanced-usage/working-with-manifests https://docs.quiltdata.com/advanced-usage/working-with-manif...
- breck 7y ago+1 for this. UC Irvine has a famous repo of datasets for machine learning research but does not have a metadata dump anywhere. I had to crawl it manually and create one (1). Would be great if you offered a single URL with CSV/JSON/other dump of your available datasets. 1. https://ohayo.computer?filename=ucimlrDemo.flow&yi=~&xi=_&data=ucimlr.datasets~_handsontable.basic https://ohayo.computer?filename=ucimlrDemo.flow&yi=~&xi=_&da...
- kevinemoore 7y agoThanks! That's really interesting feedback. We hadn't thought of that.
- kevinemoore 7y agoThe list of datasets is pretty dynamic. Would an API call work as a URL? (assuming it could return a CSV/TSV/etc.)?
- breck 7y agoYes! That is ideal.
- kevinemoore 7y agoWe'll look into that for sure!