Mosaic's open source library is excellent: Composer https://github.com/mosaicml/composer https://github.com/mosaicml/composer. * It gives you PyTorch DDP for f

Mosaic's open source library is excellent: Composer https://github.com/mosaicml/composer https://github.com/mosaicml/composer. * It gives you PyTorch DDP for free. Makes FSDP about as easy as can be, and provides best in class performance monitoring tools. https://docs.mosaicml.com/en/v0.12.1/notes/distributed_training.html#fullyshardeddataparallel-fsdp https://docs.mosaicml.com/en/v0.12.1/notes/distributed_train... Here's a nice intro to using Huggingface models: https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggingface.html https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggi... I'm just a huge fan of their developer experience. It's up there with Transformers and Datasets as the nicest tools to use.