4 ms·
Check out Determined https://github.com/determined-ai/determined https://github.com/determined-ai/determined to help manage this kind of work at scale: Determin
by etrain 5y ago
Check out Determined https://github.com/determined-ai/determined https://github.com/determined-ai/determined to help manage this kind of work at scale: Determined leverages Horovod under the hood, automatically manages cloud resources and can get you up on spot instances, T4's, etc. and will work on your local cluster as well. Gives you additional features like experiment management, scheduling, profiling, model registry, advanced hyperparameter tuning, etc.
Full disclosure: I'm a founder of the project.
- dylanbfox 5y agoInteresting. How do you guys manage spot interruptions when training on spot instances?
- etrain 5y agoUsers expose their model to our Trial API (https://docs.determined.ai/latest/topic-guides/model-definitions/trial-api.html https://docs.determined.ai/latest/topic-guides/model-definit...), the base class then implements a training loop (which can be enhanced with user-supplied callbacks, metrics, etc.) that has a whole bunch of bells and whistles. Easy distributed (multi-GPU and multi-node) training, automatic checkpointing, fault tolerance, etc. Concretely, the system is regularly taking checkpoints (which include model weights and optimizer state) and so if the spots disappear (as they do), the system has enough information to resume from where things were last checkpointed when resources become available again.
- PickleAI 5y agoThanks for going open source!
- bigbillheck 5y agoOh hey I interviewed with y'all a few years back, glad to see you're still around.