3 ms·
The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) co
by xcv123 3y ago
The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M.
It is rumoured that GPT-4 was trained on 10,000 A100 processors released in 2020. Total cost is $100M at $10k each.
Today they can buy more powerful hardware and train much larger models for the same cost.
https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing https://nvidianews.nvidia.com/news/nvidia-blackwell-platform...
https://www.cerebras.net/product-chip https://www.cerebras.net/product-chip
- anko 3y ago> The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M. Just wanted to ask the question - do you think frontier has provided more or less value to the world than gpt-4?
- xcv123 3y agoGPT-4 is not a piece of hardware. The comparison makes no sense. GPT-4 was enabled by hardware. The idea is that you use something like a Frontier to create something like a GPT-4. My point was about cost of computing. What kind of GPT can you train with a $600M computer?
- acchow 3y agoIt is mind boggling to me that it was so cheap to get GPT4. That is not even 0.1% of California HSR but providing dramatically more value.
- skybrian 3y agoNot hard to beat zero. (So far.) Maybe HSR will work out, but funding is uncertain.
- deleted 3y ago[deleted]