Azure and AWS have both offered high-bandwidth cluster options that allow scaling beyond a single server for several months now. Pretty sure MosaicML also does

Azure and AWS have both offered high-bandwidth cluster options that allow scaling beyond a single server for several months now. Pretty sure MosaicML also does this but I haven't used their offering. https://www.amazon.science/blog/scaling-to-trillion-parameter-model-training-on-aws https://www.amazon.science/blog/scaling-to-trillion-paramete...