3 ms·
No you are probably overestimating the cost by 1-2 orders of magnitude. GPT-3 probably cost under $5 million, and this model is smaller and there have been algo
by sebzim4500 4y ago
No you are probably overestimating the cost by 1-2 orders of magnitude. GPT-3 probably cost under $5 million, and this model is smaller and there have been algorithmic improvements to training transformers since then.
- krisoft 4y ago> No you are probably overestimating the cost by 1-2 orders of magnitude. You are right! Wow. Thank you for correcting me. > GPT-3 probably cost under $5 million, Is that one training run or includes all the fiddling to find the right hyperparameters? Or there aren't many of those in these training or they are not that sensitive?
- sebzim4500 4y agoI think they probably did a lot of hyperparameter searching to train the smaller models and then extrapolated for the largest model, but I'm just guessing. OpenAI had a finite amount of money when they were training GPT-3, they likely do it differently now that inference costs are significant compared to training costs.
- forgotOldLogin 4y agoIn February 2018, OpenAI signed a two-year compute contract with Google that had a $63M minimum spend. See last page of their most recent available audited financials. https://rct.doj.ca.gov/Verification/Web/Download.aspx?saveas=560291.pdf&document_id=09027b8f803a8976 https://rct.doj.ca.gov/Verification/Web/Download.aspx?saveas...
- dmix 4y agoSo do they estimate how much computing power/time they will need and then find some upper tier minimum $ amount to get the maximum discount possible or getting a certain resource availability commitment from Google? That's an interesting accounting problem.