3 ms·
3500 of them trained GPT-3 in 11 minutes. That number is still worth remarking.
by messe 3y ago
3500 of them trained GPT-3 in 11 minutes. That number is still worth remarking.
- reaperman 3y agoIt’s remarkable that one of these could train GPT-3 in under a month. However, good to note that its the price is that of twenty 4090’s.
- ClassyJacket 3y agoCould it actually? Or do you need the memory size of thousands of them running in parallel?
- reaperman 3y agoYes I worded that extremely poorly.
- cavisne 3y agoOne interesting thing is if the model can’t fit into GPU memory (sharded across multiple chips) it would be much slower. So one chip would take more than a month. That’s why these clusters are so valuable, even with Nvidias margins they are still cheaper than using less compute for longer.
- visarga 3y agotitle says: "a massive GPT-3-based benchmark" that means they might pass a few batches through a GPT-3 sized random init network and time it
- layoric 3y agoGPT3 benchmark, as others have said this means it would be ~46 hours to train GPT, still impressive! Also, going by other comments here, ~3500 of these or 13-14 DGX GH200s at ~$10m each means we are talking about ~$135m worth of compute here. Still very impressive but holy hell that is a lot of money worth of compute hardware.