4 ms·
This latest model was trained on a distributed multi-node cluster of 8 servers each with 8 NVIDIA A100 GPUs. Need a lot of GPU memory when training with long se
by rushingcreek 4y ago
This latest model was trained on a distributed multi-node cluster of 8 servers each with 8 NVIDIA A100 GPUs. Need a lot of GPU memory when training with long sequence length.
- alchemist1e9 4y agoFor how long do you know?
- EastSmith 4y agoLooks like a single NVIDIA A100 costs $23,667.29 [1]? So 8 servers X 8 cards X $23,667.29 means the whole setup costs $1,514,706.56? https://www.dell.com/en-us/shop/nvidia-ampere-a100-pcie-300w-80gb-passive-double-wide-full-height-gpu-customer-install/apd/490-bhbk/graphic-video-cards https://www.dell.com/en-us/shop/nvidia-ampere-a100-pcie-300w...
- rushingcreek 4y agoAt list prices, perhaps. Importantly, we don't own the hardware. H100s are coming soon and they are much faster.
- Jayakumark 4y agoWas it trained on Common Crawl Dataset or custom scraping ?