3 ms·
We recently trained GPT-3 (SMALL) at work on our GPU cluster for fun, took 4 days across a couple dozen machines... Millions of dollars in CAPEX and OPEX just
by GaryNumanVevo 6y ago
We recently trained GPT-3 (SMALL) at work on our GPU cluster for fun, took 4 days across a couple dozen machines...
Millions of dollars in CAPEX and OPEX just for one model
- stainforth 6y agoYou're saying your project costed millions of dollars, or the big boys' projects did?
- shmageggy 6y agoIf "4 days across a couple dozen machines" cost millions, something is very wrong.
- deleted 6y ago[deleted]
- T-A 6y agoNot if it was a couple dozen of these machines: https://www.hardwarezone.com.sg/tech-news-nvidia-dgx-a100-supercomputer-super-performance-fight-covid-19 https://www.hardwarezone.com.sg/tech-news-nvidia-dgx-a100-su...
- GaryNumanVevo 6y agoRunning your own DC is quite expensive with GPU hardware. One DGX-2 is $400k and draws something like 24kW.
- rajnathani 6y ago> draws something like 24 kW. That number is off. The DGX-2 consumes 10 kW at peak [0] and the DGX-2H consumes 12 kW at peak [1]. [0] https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/dgx-2/dgx-2-print-datasheet-738070-nvidia-a4-web-uk.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent... [1] https://www.nvidia.com/content/dam/en-zz/es_em/Solutions/Data-Center/dgx-2/nvidia-dgx-2h-datasheet.pdf https://www.nvidia.com/content/dam/en-zz/es_em/Solutions/Dat...
- ImprobableTruth 6y agoI'm curious: Did you get comparable results to openai? I know a few people tried to train GPT-2 themselves (before it was openly released) and their results were quite inferior.