3 ms·
It’s remarkable that one of these could train GPT-3 in under a month. However, good to note that its the price is that of twenty 4090’s.
by reaperman 3y ago
It’s remarkable that one of these could train GPT-3 in under a month. However, good to note that its the price is that of twenty 4090’s.
- ClassyJacket 3y agoCould it actually? Or do you need the memory size of thousands of them running in parallel?
- reaperman 3y agoYes I worded that extremely poorly.
- cavisne 3y agoOne interesting thing is if the model can’t fit into GPU memory (sharded across multiple chips) it would be much slower. So one chip would take more than a month. That’s why these clusters are so valuable, even with Nvidias margins they are still cheaper than using less compute for longer.