3 ms·
I learned some things. GTP-3 is so complex the model requires large cloud computing resources to run. Ergo, it is also very expensive to run. Assumption: Blee
by mensetmanusman 6y ago
I learned some things.
GTP-3 is so complex the model requires large cloud computing resources to run. Ergo, it is also very expensive to run.
Assumption: Bleeding edge AI will require tens of millions of dollars of computation before new network architectures fall out of state space. After this, the models can be pruned to be ran by mere mortals.
If this is true, OpenAI will not be able to move to the next level without partnering with very wealthy institutions a la Microsoft.
If this is true, those calling for OpenAI to not monetize intermittent progress are essentially preventing next generation discovery, unless they have alternative monetization ideas to generate 8 figures for research.
- choppaface 6y agoThere will always be models and research that require proprietary hardware and resources and thus cannot be “democratized.” PageRank required a ton of hardware and an index. Non-industry researchers will always be priced out of something. A key concern here with regards to the ethics-of-AI issue is that last year OpenAI refused to release GPT-2 because it was too ‘dangerous’. This year, GPT-3 is suddenly a revenue-generating Microsoft product. The Gradient article linked is one of the more diplomatic ways of calling BS on OpenAI’s strategy. Financial interests taint research in subtle ways, it has for decades, and OpenAI employees being paid $1m in cash salary need reminders of this fact. https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/
- anoncareer0212 6y agoGiven the worst-case assumptions of people arguing the cloud is required[1], it's a one-time investment of $60,000, $200,000 in the absolute worst case. [1] https://news.ycombinator.com/item?id=24601264 https://news.ycombinator.com/item?id=24601264
- exged 6y agoThat's for model inference. For training OpenAI said they used around 3000 PetaFLOPS / days on the largest GPT-3 model. That translates to about 300 Nvidia A100 GPUs if you want training to finish in a month (any slower and your researchers are not going to be able to make much progress). A system like that would cost at least $5M, probably more like $10M.
- lopmotr 6y agoIs that unit PetaFLOPS/day right? I think it should be PetaFLOPS-day which has dimensions of FLOP (total number of operations), rather than FLOP/time^2. The cost wouldn't be the cost of the hardware because it still exists afterwards. You'd have to discount it for the amount of time it was in use.
- 2sk21 6y agoSurely $10 million should be well within the spending abilities of a fair number of tech people? Many universities now have fairly large computing clusters as well.
- claudeganon 6y agoPeople give more than this to get their lackluster children into the Ivy League.
- PeterStuer 6y agoSure there might be challenges and opportunities on the bleeding edge. However, in this case, this opportunity was so orthogonal to their original mission statement it should not come as surprise t does more than raise eyebrows.
- ralfn 6y ago>Ergo, it is also very expensive to run Wether thats too expensive depends on what you are doing, and how much what you are doing creates value.More disturbingly is that the whole approach seems to scale linearly in quality based on the amount of training. This implies that the NLP market at least, will not be conquored by start-ups from somebody's garage, but will be owned by whoever already had a shit load of money. Whoever gets more funding will have the best model. Not arguing that people involved don't have a lot of skill, but within this space, its funding (not skill) that will determine who wins the market. So, unless you can get more funding than the other guys, don't even try the NLP space. >If this is true, those calling for OpenAI to not monetize intermittent progress are essentially preventing next generation discovery, unless they have alternative monetization ideas to generate 8 figures for research I suspect they will share the models directly with select customers (with too much money), because the negotation position of these counterparties will be different. The results are easy to replicate with a lot of money. So if you have enough money to do so, your price negotation with them would be more like 'ill pay you 1/3 of that price to liscense your model and save me the time'. If you dont have enough money to pay the electricity bill to train this much data, however, you can be forced to comply with this bussiness model. Obviously, there will be competitors. Obviously the big boys will likely try to replicate these results (and be succesfull at it). The hope is one of them just open-sources a 'good-enough' model. Its a bit like Colombus 'discovering America'. Once you know its there the risk/reward of trying to go there drastically changes.