5 ms·
Hey, congrats :) It's good to see more tools in this space. Question - how does it handle datasets and data management in general? Let's say we have a large amo
by shcheklein 4y ago
Hey, congrats :) It's good to see more tools in this space. Question - how does it handle datasets and data management in general? Let's say we have a large amount of files on S3. Would it my task to copy them first as an initial step?
- cheptsov 4y agoThank you! That's a great question. First and foremost, dstack treats artifacts are 1st class citizens. Here's the basic way of using them: A workflow may produce output files to a local folder. In your workflow declaration, you can mark what folders to treat as output artifacts. Then, dstack would save output artifacts automatically, and you'll be able to reuse them via the unique name of the run, or a user tag assigned to this run. All artifacts are stored in the S3 bucket that is configured for dstack. dstack is capable of syncing artifacts at start//end of the workflow or mount artifact folders via FUSE (of course if that is needed). Each artifact is stored using the following path: <s3 bucket>/artifacts/<run name>/<job name> A run can have multiple jobs, e.g. if it's a distributed workflow. In future, we also think of providing a high-level Python API for accessing/storing artifacts. Please share your thoughts and feedback!
- cheptsov 4y agoYes, by default, it will copy the files. Or you could also tell dstack to use FUSE. As I said in the comment above, we think of providing a high-level Python API. THis API will allow to avoid copying the artifacts as an initial step. But of course would love to hear what you think! ;-)