3 ms·
"Combined with the additional memory on H100 and the faster NVLink 4 I/O, and NVIDIA claims that a large cluster of GPUs can train a transformer up to 9x faster
by rafaelero 5y ago
"Combined with the additional memory on H100 and the faster NVLink 4 I/O, and NVIDIA claims that a large cluster of GPUs can train a transformer up to 9x faster, which would bring down training times on today’s largest models down to a more reasonable period of time, and make even larger models more practical to tackle."
Looking good.
- Melatonic 5y agoWe have needed wide use of NVlink or something like it for a long time now......heres to hoping mobo manufacturers actually widely implement it!
- learndeeply 5y agoThe open standard version of NVLink is CXL. They're available in latest gen CPUs.
- Melatonic 5y agoInteresting - I did not know that. Don't we also need motherboard manufacturers though to more widely implement the hardware required? It has been awhile since I have read about NVlink to be fair
- ml_hardware 5y agoThe 9x speedup is a bit inflated... it's measured at a reference point of ~8k GPUs, on a workload that the A100 cluster is particularly bad at. When measured at smaller #s of GPUs which are more realistic, the speedup is somewhere between 3.5x - 6x. See the GTC Keynote video at 38:50: https://youtu.be/39ubNuxnrK8?t=2330 https://youtu.be/39ubNuxnrK8?t=2330 Based on hardware specs alone, I think that training transformers with FP8 on H100 systems vs. FP16 on A100 systems should only be 3-4x faster. Definitely looking forward to external benchmarks over the coming months...
- deleted 5y ago[deleted]