4 ms·

Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1

Why isn't there a framework that does all this automatically for you? I tried torch FSDP but it only managed to increase the memory to something like 150% of 1 GPU. I eventually ended up sharding my model manually with .cuda() and .to() which works much better, but now I am limited to one module on one GPU and I would like to expand even more, and that would mean spinning up more nodes and splitting the model over how many GPUs manually. I would be interested if anyone knows of a framework that manages this automatically and just works. EDIT: BTW I am talking about model sharding not data parallelism which works very well with DDP.

arcanus 4y ago
It's a fair question. Nvidia's NCCL and AMD's RCCL provide parallelism constructs that really are hidden at the framework level (such as PyT). However, I don't think that you would want to hide model, data, or tensor parallelism. It's too important a consideration for performance and training convergence impact. At least in scientific computing, I've never observed effective means of automatic parallelism expressed across many nodes despite decades of research. I'm not optimistic this will be effective anytime soon.
minimaxir 4y ago
DeepSpeed became popular soon after this post was originally published and is natively supported by many PyTorch training frameworks. https://www.deepspeed.ai https://www.deepspeed.ai https://www.deepspeed.ai/training/ https://www.deepspeed.ai/training/
- dauertewigkeit 4y ago
  I tried that as well, but maybe I did not use it correctly. I did not see the full sharding that I was hoping for. I only saw results similiar to FSDP.
  - cma 4y ago
    How about flexflow? https://huggingface.co/transformers/v4.9.2/parallelism.html#flexflow https://huggingface.co/transformers/v4.9.2/parallelism.html#...
guardiantesla 4y ago
>Why isn't there a framework that does all this automatically for you? Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times https://www.mosaicml.com/blog/supercharge-training-composer https://www.mosaicml.com/blog/supercharge-training-composer Also worth checking out their “training from scratch” blog posts. Training StableDiffusion: https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-costs-160k https://www.mosaicml.com/blog/training-stable-diffusion-from... Training GPT-3: https://www.mosaicml.com/blog/billion-parameter-gpt-training-made-easy https://www.mosaicml.com/blog/billion-parameter-gpt-training...
- NerdyDrone 4y ago
  Mosaic's open source library is excellent: Composer https://github.com/mosaicml/composer https://github.com/mosaicml/composer. * It gives you PyTorch DDP for free. Makes FSDP about as easy as can be, and provides best in class performance monitoring tools. https://docs.mosaicml.com/en/v0.12.1/notes/distributed_training.html#fullyshardeddataparallel-fsdp https://docs.mosaicml.com/en/v0.12.1/notes/distributed_train... Here's a nice intro to using Huggingface models: https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggingface.html https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggi... I'm just a huge fan of their developer experience. It's up there with Transformers and Datasets as the nicest tools to use.
  - kanyethegreat 4y ago
    > Transformers and Datasets Are these specific libraries?
    - swyx 4y ago
      https://github.com/huggingface/datasets https://github.com/huggingface/datasets https://github.com/huggingface/transformers https://github.com/huggingface/transformers
sandkoan 4y ago
Might this be what you're looking for: https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals ?
option 4y ago
There is - https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/intro.html https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en... Supports data, tensor, pipeline, sequence parallelisms, activation checkpointing, distributed optimizers, fused kernels and more.
amelius 4y ago
> Why isn't there a framework that does all this automatically for you? Question: could this be implemented in PyTorch in an opaque way? Or would it require changes to its API?
atty 4y ago
Beyond the other answers, I’ll point out that pytorch is developing tools that will make doing this work by hand or implementing in a framework much easier. They’re building a native DTensor implementation and testing out SPMD-style distributed models with pipelining. DTensor is in torch.distributed, and the SPMD code is in the repo called Tau under the pytorch org on github.
buildbot 4y ago
Any framework that "just works" tends to not work when some small change is needed or a new model with new data/compute roofline comes out.

arcanus 4y ago

It's a fair question. Nvidia's NCCL and AMD's RCCL provide parallelism constructs that really are hidden at the framework level (such as PyT). However, I don't think that you would want to hide model, data, or tensor parallelism. It's too important a consideration for performance and training convergence impact. At least in scientific computing, I've never observed effective means of automatic parallelism expressed across many nodes despite decades of research. I'm not optimistic this will be effective anytime soon.

minimaxir 4y ago

DeepSpeed became popular soon after this post was originally published and is natively supported by many PyTorch training frameworks. https://www.deepspeed.ai https://www.deepspeed.ai https://www.deepspeed.ai/training/ https://www.deepspeed.ai/training/

dauertewigkeit 4y ago
I tried that as well, but maybe I did not use it correctly. I did not see the full sharding that I was hoping for. I only saw results similiar to FSDP.
- cma 4y ago
  How about flexflow? https://huggingface.co/transformers/v4.9.2/parallelism.html#flexflow https://huggingface.co/transformers/v4.9.2/parallelism.html#...

guardiantesla 4y ago

>Why isn't there a framework that does all this automatically for you? Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times https://www.mosaicml.com/blog/supercharge-training-composer https://www.mosaicml.com/blog/supercharge-training-composer Also worth checking out their “training from scratch” blog posts. Training StableDiffusion: https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-costs-160k https://www.mosaicml.com/blog/training-stable-diffusion-from... Training GPT-3: https://www.mosaicml.com/blog/billion-parameter-gpt-training-made-easy https://www.mosaicml.com/blog/billion-parameter-gpt-training...

NerdyDrone 4y ago
Mosaic's open source library is excellent: Composer https://github.com/mosaicml/composer https://github.com/mosaicml/composer. * It gives you PyTorch DDP for free. Makes FSDP about as easy as can be, and provides best in class performance monitoring tools. https://docs.mosaicml.com/en/v0.12.1/notes/distributed_training.html#fullyshardeddataparallel-fsdp https://docs.mosaicml.com/en/v0.12.1/notes/distributed_train... Here's a nice intro to using Huggingface models: https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggingface.html https://docs.mosaicml.com/en/v0.12.1/examples/finetune_huggi... I'm just a huge fan of their developer experience. It's up there with Transformers and Datasets as the nicest tools to use.
- kanyethegreat 4y ago
  > Transformers and Datasets Are these specific libraries?
  - swyx 4y ago
    https://github.com/huggingface/datasets https://github.com/huggingface/datasets https://github.com/huggingface/transformers https://github.com/huggingface/transformers

sandkoan 4y ago

Might this be what you're looking for: https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals ?

option 4y ago

There is - https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/intro.html https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en... Supports data, tensor, pipeline, sequence parallelisms, activation checkpointing, distributed optimizers, fused kernels and more.

amelius 4y ago

> Why isn't there a framework that does all this automatically for you? Question: could this be implemented in PyTorch in an opaque way? Or would it require changes to its API?

atty 4y ago

Beyond the other answers, I’ll point out that pytorch is developing tools that will make doing this work by hand or implementing in a framework much easier. They’re building a native DTensor implementation and testing out SPMD-style distributed models with pipelining. DTensor is in torch.distributed, and the SPMD code is in the repo called Tau under the pytorch org on github.

buildbot 4y ago

Any framework that "just works" tends to not work when some small change is needed or a new model with new data/compute roofline comes out.