3 ms·

These are order-of-magnitude increases. If it costs $5m to train GPT-3, which is 100x more compute than GPT-2, then it may cost $500m to train GPT-4, and $50b

These are order-of-magnitude increases. If it costs $5m to train GPT-3, which is 100x more compute than GPT-2, then it may cost $500m to train GPT-4, and $50b to train GPT-5. This is what is meant by economically (not to mention environmentally) unsustainable.

freeone3000 6y ago
Environment aside (I don't even think the CURRENT rate of training is environmentally safe) -- I see no inherent problem with a 100x increase in cost per step holding anybody back. Once it's trained, you can run it much cheaper. Who's to say $50 billion of value can't be extracted from GPT-5?
- AnimalMuppet 6y ago
  > Who's to say $50 billion of value can't be extracted from GPT-5? Perhaps it can. (Though the number of companies that can afford to train it is rather small.) But can $5 trillion be extracted from GPT-6? Even if it can, who can afford to train it?
  - dane-pgp 6y ago
    If GPT-5 is capable of creating a cost-efficient implementation of GPT-6, then maybe we don't need to worry about the extrapolated $5 trillion price tag.
- saddlerustle 6y ago
  Using electricity is not inherently damaging to the environment. Very low cost and high power zero-carbon generation sources exist (hydro and nuclear). For scale also keep in mind that all the datacenters in the world still use much less power than is used for smelting aluminium.
  - jdietrich 6y ago
    Also, processors turn electricity into heat - if you can find a useful way of harnessing that heat, the computation is practically free. Deep learning is fairly latency-insensitive and is almost never mission-critical, so it's not inconceivable to imagine that the boiler rooms of apartment buildings and office blocks could become mini-DCs.
    - p1esk 6y ago
      Also, once there are billions of value to be extracted, specialized chips will appear just to train those models (e.g. wafer scale analog processors, could easily be 3 orders of magnitude more efficient than GPUs/TPUs).
    - fock 6y ago
      oh wait, decentralized, publicly controlled infra. What was the cloud about exactly?

freeone3000 6y ago

Environment aside (I don't even think the CURRENT rate of training is environmentally safe) -- I see no inherent problem with a 100x increase in cost per step holding anybody back. Once it's trained, you can run it much cheaper. Who's to say $50 billion of value can't be extracted from GPT-5?

AnimalMuppet 6y ago
> Who's to say $50 billion of value can't be extracted from GPT-5? Perhaps it can. (Though the number of companies that can afford to train it is rather small.) But can $5 trillion be extracted from GPT-6? Even if it can, who can afford to train it?
- dane-pgp 6y ago
  If GPT-5 is capable of creating a cost-efficient implementation of GPT-6, then maybe we don't need to worry about the extrapolated $5 trillion price tag.
saddlerustle 6y ago
Using electricity is not inherently damaging to the environment. Very low cost and high power zero-carbon generation sources exist (hydro and nuclear). For scale also keep in mind that all the datacenters in the world still use much less power than is used for smelting aluminium.
- jdietrich 6y ago
  Also, processors turn electricity into heat - if you can find a useful way of harnessing that heat, the computation is practically free. Deep learning is fairly latency-insensitive and is almost never mission-critical, so it's not inconceivable to imagine that the boiler rooms of apartment buildings and office blocks could become mini-DCs.
  - p1esk 6y ago
    Also, once there are billions of value to be extracted, specialized chips will appear just to train those models (e.g. wafer scale analog processors, could easily be 3 orders of magnitude more efficient than GPUs/TPUs).
  - fock 6y ago
    oh wait, decentralized, publicly controlled infra. What was the cloud about exactly?