4 ms·
Do you store the datasets as tar/zip archives on S3, or do you have some way of representing how a collection of items goes together to form a dataset?
by jononor 8y ago
Do you store the datasets as tar/zip archives on S3, or do you have some way of representing how a collection of items goes together to form a dataset?
- kevinemoore 8y agoQuilt co-founder here, no, Quilt doesn’t use tar or zip. Each package version has a manifest that specifies the set of items it contains. Each item in the collection is stored and transported separately. Items are identified by their hash and stored once even if used in more than one package version.
- mlthoughts2018 8y agoFor some data sets we use an archive format and the corresponding script unpackages the data. For others, we have all individual files in S3. Using archives improves download speed, and sometimes we provide both, so some narrowly-defined data sets benefit from the archive file, while others can pick and choose specific samples to include or exclude. One of the nice things about the simple script approach is that the dataset can be defined however you like. Whatever the script retrieves and unpacks for you, that is the data set of that script. It could be a superset or subset of other data, and intersection of samples with a certain property from other data. As long as that definition is treated as immutable, and the backing data is immutable, it defines a specific collection of items however you want.