4 ms·

>Why isn't there a framework that does all this automatically for you? Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most cu

>Why isn't there a framework that does all this automatically for you? Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times https://www.mosaicml.com/blog/supercharge-training-composer https://www.mosaicml.com/blog/supercharge-training-composer Also worth checking out their “training from scratch” blog posts. Training StableDiffusion: https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-costs-160k https://www.mosaicml.com/blog/training-stable-diffusion-from... Training GPT-3: https://www.mosaicml.com/blog/billion-parameter-gpt-training-made-easy https://www.mosaicml.com/blog/billion-parameter-gpt-training...

NerdyDrone 4y ago
Mosaic's open source library is excellent: Composer https://github.com/mosaicml/composer https://github.com/mosaicml/composer. * It gives you PyTorch DDP for free. Makes FSDP about as easy as can be, and provides best in class performance monitoring tools. https://docs.mosaicml.com/en/v0.12.1/notes/distributed_training.html#fullyshardeddataparallel-fsdp https://docs.mosaicml.com/en/v0.12.1/notes/distributed_train... Here's a nice intro to using Huggingface models: https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggingface.html https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggi... I'm just a huge fan of their developer experience. It's up there with Transformers and Datasets as the nicest tools to use.
- kanyethegreat 4y ago
  > Transformers and Datasets Are these specific libraries?
  - swyx 4y ago
    https://github.com/huggingface/datasets https://github.com/huggingface/datasets https://github.com/huggingface/transformers https://github.com/huggingface/transformers