5 ms·
This is really cool to see; I'm glad karpathy shared the work. The internet at large has been really good at taking opaque machine learning research and elucid
by fpgaminer 6y ago
This is really cool to see; I'm glad karpathy shared the work.
The internet at large has been really good at taking opaque machine learning research and elucidating the details and recreating the results. I've seen a few posts/repositories/etc doing that for GPT-3 as well but man, 175B parameters is just so far out of reach for hobbyists. It's really a shame.
In time further research will likely make language models more efficient to where something GPT-3-like can be trained at the hobbyist level. Probably a blended model like SHA-RNN or something with dedicated memory in its architecture, so the model isn't burning precious weights on remembering e.g. Lincoln's birthday. In the meantime though it makes me sad that something as impressive as GPT-3 is solely the toy of corporations.
- GaryNumanVevo 6y agoWe recently trained GPT-3 (SMALL) at work on our GPU cluster for fun, took 4 days across a couple dozen machines... Millions of dollars in CAPEX and OPEX just for one model
- stainforth 6y agoYou're saying your project costed millions of dollars, or the big boys' projects did?
- shmageggy 6y agoIf "4 days across a couple dozen machines" cost millions, something is very wrong.
- deleted 6y ago[deleted]
- T-A 6y agoNot if it was a couple dozen of these machines: https://www.hardwarezone.com.sg/tech-news-nvidia-dgx-a100-supercomputer-super-performance-fight-covid-19 https://www.hardwarezone.com.sg/tech-news-nvidia-dgx-a100-su...
- GaryNumanVevo 6y agoRunning your own DC is quite expensive with GPU hardware. One DGX-2 is $400k and draws something like 24kW.
- rajnathani 6y ago> draws something like 24 kW. That number is off. The DGX-2 consumes 10 kW at peak [0] and the DGX-2H consumes 12 kW at peak [1]. [0] https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/dgx-2/dgx-2-print-datasheet-738070-nvidia-a4-web-uk.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent... [1] https://www.nvidia.com/content/dam/en-zz/es_em/Solutions/Data-Center/dgx-2/nvidia-dgx-2h-datasheet.pdf https://www.nvidia.com/content/dam/en-zz/es_em/Solutions/Dat...
- ImprobableTruth 6y agoI'm curious: Did you get comparable results to openai? I know a few people tried to train GPT-2 themselves (before it was openly released) and their results were quite inferior.