3 ms·
> Nvidia demonstrated this back in 2021. Because Nvidia created a supercomputer with A100, with lot of focus for networking. Cloud providers don't give that op
by YetAnotherNick 3y ago
> Nvidia demonstrated this back in 2021.
Because Nvidia created a supercomputer with A100, with lot of focus for networking. Cloud providers don't give that option.
- haldujai 3y agoAzure and AWS have both offered high-bandwidth cluster options that allow scaling beyond a single server for several months now. Pretty sure MosaicML also does this but I haven't used their offering. https://www.amazon.science/blog/scaling-to-trillion-parameter-model-training-on-aws https://www.amazon.science/blog/scaling-to-trillion-paramete...