3 ms·
The cost of developing GPT-2 and GPT-3 was on the order of millions of dollars, well within the budget of most tech organizations. See https://arxiv.org/abs/200
by cs702 1y ago
The cost of developing GPT-2 and GPT-3 was on the order of millions of dollars, well within the budget of most tech organizations. See https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 for the total compute invested in them. OpenAI had raised only a few tens of millions in donations at the time, as a non-profit organization.
The increase in performance of computation has been happening for so many decades now that it's been given names like "Moore's Law." People like Hans Moravec predicted way back in the 1980's that the cost of compute would continue to decline and become cheap enough for AGI by the 2020's or 2030's. That's half a century ago!
- chermi 1y agoFair point, the first few gens weren't that expensive. And like I said I'm certain the scaling would've been discovered soon with some time lag. But the transformer paper was 2017, right? Just from a benefit -to- society perspective, assuming LLMs are a net positive in terms of productivity, perhaps reducing time to drug approval or improving government efficiency (1).. Isn't getting there just one year earlier worth it, if the gains really are that big? We could be talking lives saved via faster approvals or more efficient spending. My point is that a private company made it happen faster, as evidenced by them doing it first. A good thing sooner is valuable. (1) I'm convinced at the very least LLMs can feasibly speed up paperwork.
- loglog 1y agoI believe that LLMs are detrimental to government efficiency, because they distract from solving the underlying problems (such as porkishly complex laws or a lack of unique identifiers).