4 ms·

Was not aware of Quilt for hosting datasets. Is it the go-to in this area? What other alternatives are there?

eindiran 8y ago
You can use AWS to host open datasets: https://aws.amazon.com/opendata/public-datasets/ https://aws.amazon.com/opendata/public-datasets/ These are some other people working in roughly the same space: http://datproject.org/ http://datproject.org/ http://www.pachyderm.io/ http://www.pachyderm.io/ But it does seem like Quilt is a go-to, if you are looking for a "Github for data" host.
- jmaxfield 8y ago
  I use Quilt pretty much daily and while I like AWS open datasets I don't think it is as actively developed on as Quilt is. DAT project on the other hand I really do like as a way to simply transfer large amounts of data between contributors, that said, if you are just trying to get data out there and have people use it freely for their own work I think Quilt presents the solution due to searchable and easily understood python (and I think an R repo) usage of datasets.
- jdoliner 8y ago
  Pachyderm founder here. We're not really a data hosting provider, although we may offer that in the future. Right now Pachyderm is more like intranet data hosting for companies. You have to spin up your own Kubernetes cluster and deploy Pachyderm on it. It's also not normally used to download data onto your local machine for processing because it has its own computation layer which allows you to run code at scale and tracks the provenance of the data to keep things reproducible.
shoyer 8y ago
A couple of publicly available alternatives I'm aware of include: - Girder: http://girder.readthedocs.io http://girder.readthedocs.io - Intake: https://github.com/ContinuumIO/intake https://github.com/ContinuumIO/intake I haven't used any of these, but I agree that the idea is quite compelling.
casegold 8y ago
Quilt has been great for CLI versioning while preparing large datasets
rryan 8y ago
dat: https://datproject.org/ https://datproject.org/
guenp 8y ago
I've tried dat (https://datproject.org/ https://datproject.org/) and git lfs (https://git-lfs.github.com/ https://git-lfs.github.com/) but so far have found quilt to be easiest to use & best fitting to my use case (experimental physics characterization experiments).
DTE 8y ago
I have been researching data orchestration/versioning tools for a long time and have been following the Quilt guys closely. It is definitely one of the more powerful tools in the ML/AI engineer's toolbox and solves a huge problem that almost everyone runs in to right out the gate. It's still early days in this space but Quilt gets a lot of things right and I'm super excited to see this product develop. Full disclosure: I run Paperspace (https://www.paperspace.com https://www.paperspace.com) and am working with the Quilt team to integrate their tools in to our platform.
mlthoughts2018 8y ago
Just use AWS S3 (or similar) and shell scripts. My team uses a git repository named something like "data-packages", which is nothing but a collection of shell scripts with the name <dataset>.sh, that perform the necessary download and extraction steps to get a dataset from S3. Data sets are immutable by convention, so any changes to a data set requires you to provide a totally new shell script. That script could download an older data set and then mutate it if you don't want to maintain large copies of big data sets, but the older data set itself is not permitted to be mutated on S3. My team has found this drastically easier than Quilt, and we do a ton of stuff with reproducible environments in Docker, creating Makefiles to reproduce exact model training with the exact same data, etc. We probably hit just about every case there is (huge models, small models, models where we'd like to train separately or collectively on a bunch of different benchmark data sets, in-house data sets, models that need to be refreshed with new data in pipelines, etc.) So far, Quilt has not been competitive with a simple repo of shell scripts for us, in terms of ease of use or effectiveness in maintaining different packages of data. The other super nice thing is that when people start out on new models or experiments, we already have our in-house maintained copies of a bunch of academic data sets, private data sets, etc., and you can throw together an incredibly simple Dockerfile or Makefile that uses the appropriate script. It's just one or two lines of shell code and voila, you have an environment with the dataset you want. Check that into git and now your experiment is immediately reproducible from day one. We've found this to dramatically increase the amount of code review that researchers engage in for checking their statistical methodology and sanity checking their intended models or experiments. With Quilt, you have the extra issue of versioning (rather than harshly enforcing all data sets to be immutable ... even just adding one more training example to the data set means you must provide a new shell script that downloads the old data, injects your lone additional sample, and has a documentation entry about exactly what it is doing), as well as the overhead of using yet another tool instead of super standard shell scripts. For me, any of the tools that pop up attempting to be like conda-forge but for data packages is sort of like taking a gatling gun to a problem that can be solved with a hammer.
- jononor 8y ago
  Do you store the datasets as tar/zip archives on S3, or do you have some way of representing how a collection of items goes together to form a dataset?
shauni 8y ago
Has anyone tried out the option of self-hosting Quilt registries? I really like the idea of Quilt, although I am worried that my network bandwidth would be an issue for 10-100GB datasets...

You can use AWS to host open datasets: https://aws.amazon.com/opendata/public-datasets/ https://aws.amazon.com/opendata/public-datasets/ These are some other people working in roughly the same space: http://datproject.org/ http://datproject.org/ http://www.pachyderm.io/ http://www.pachyderm.io/ But it does seem like Quilt is a go-to, if you are looking for a "Github for data" host.

jmaxfield 8y ago
I use Quilt pretty much daily and while I like AWS open datasets I don't think it is as actively developed on as Quilt is. DAT project on the other hand I really do like as a way to simply transfer large amounts of data between contributors, that said, if you are just trying to get data out there and have people use it freely for their own work I think Quilt presents the solution due to searchable and easily understood python (and I think an R repo) usage of datasets.
jdoliner 8y ago
Pachyderm founder here. We're not really a data hosting provider, although we may offer that in the future. Right now Pachyderm is more like intranet data hosting for companies. You have to spin up your own Kubernetes cluster and deploy Pachyderm on it. It's also not normally used to download data onto your local machine for processing because it has its own computation layer which allows you to run code at scale and tracks the provenance of the data to keep things reproducible.

shoyer 8y ago

A couple of publicly available alternatives I'm aware of include: - Girder: http://girder.readthedocs.io http://girder.readthedocs.io - Intake: https://github.com/ContinuumIO/intake https://github.com/ContinuumIO/intake I haven't used any of these, but I agree that the idea is quite compelling.

casegold 8y ago

Quilt has been great for CLI versioning while preparing large datasets

rryan 8y ago

dat: https://datproject.org/ https://datproject.org/

guenp 8y ago

I've tried dat (https://datproject.org/ https://datproject.org/) and git lfs (https://git-lfs.github.com/ https://git-lfs.github.com/) but so far have found quilt to be easiest to use & best fitting to my use case (experimental physics characterization experiments).

DTE 8y ago

I have been researching data orchestration/versioning tools for a long time and have been following the Quilt guys closely. It is definitely one of the more powerful tools in the ML/AI engineer's toolbox and solves a huge problem that almost everyone runs in to right out the gate. It's still early days in this space but Quilt gets a lot of things right and I'm super excited to see this product develop. Full disclosure: I run Paperspace (https://www.paperspace.com https://www.paperspace.com) and am working with the Quilt team to integrate their tools in to our platform.

mlthoughts2018 8y ago

Just use AWS S3 (or similar) and shell scripts. My team uses a git repository named something like "data-packages", which is nothing but a collection of shell scripts with the name <dataset>.sh, that perform the necessary download and extraction steps to get a dataset from S3. Data sets are immutable by convention, so any changes to a data set requires you to provide a totally new shell script. That script could download an older data set and then mutate it if you don't want to maintain large copies of big data sets, but the older data set itself is not permitted to be mutated on S3. My team has found this drastically easier than Quilt, and we do a ton of stuff with reproducible environments in Docker, creating Makefiles to reproduce exact model training with the exact same data, etc. We probably hit just about every case there is (huge models, small models, models where we'd like to train separately or collectively on a bunch of different benchmark data sets, in-house data sets, models that need to be refreshed with new data in pipelines, etc.) So far, Quilt has not been competitive with a simple repo of shell scripts for us, in terms of ease of use or effectiveness in maintaining different packages of data. The other super nice thing is that when people start out on new models or experiments, we already have our in-house maintained copies of a bunch of academic data sets, private data sets, etc., and you can throw together an incredibly simple Dockerfile or Makefile that uses the appropriate script. It's just one or two lines of shell code and voila, you have an environment with the dataset you want. Check that into git and now your experiment is immediately reproducible from day one. We've found this to dramatically increase the amount of code review that researchers engage in for checking their statistical methodology and sanity checking their intended models or experiments. With Quilt, you have the extra issue of versioning (rather than harshly enforcing all data sets to be immutable ... even just adding one more training example to the data set means you must provide a new shell script that downloads the old data, injects your lone additional sample, and has a documentation entry about exactly what it is doing), as well as the overhead of using yet another tool instead of super standard shell scripts. For me, any of the tools that pop up attempting to be like conda-forge but for data packages is sort of like taking a gatling gun to a problem that can be solved with a hammer.

jononor 8y ago
Do you store the datasets as tar/zip archives on S3, or do you have some way of representing how a collection of items goes together to form a dataset?

shauni 8y ago

Has anyone tried out the option of self-hosting Quilt registries? I really like the idea of Quilt, although I am worried that my network bandwidth would be an issue for 10-100GB datasets...