There is - https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/intro.html https://docs.nvidia.com/deeplearning/nemo/user-guide

There is - https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/intro.html https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en... Supports data, tensor, pipeline, sequence parallelisms, activation checkpointing, distributed optimizers, fused kernels and more.