5 ms·

The premise of this whole article is that once general purpose computing can do what the GPUs can then demand for GPUs will drop. That is a fundamentally flawed

The premise of this whole article is that once general purpose computing can do what the GPUs can then demand for GPUs will drop. That is a fundamentally flawed assumption. The ability to parallelize operations using a GPU will always be available and GPU development will continue. Hardware tech (process nodes, etc) that improves CPUs will also improve GPUs. Maybe we will reach a peak demand, but not until individual GPUs and CPUs are in millisecond token inference range. And that won't happen for a long time. The author is erroneously conflating GPU and ASIC development. To be clear, I agree that LLMs are not anywhere close to AGI and I don't think they ever will be (just a component). But that doesn't mean they aren't useful enough to chew up a lot of compute for the foreseeable future.

HarHarVeryFunny 2y ago
> To be clear, I agree that LLMs are not anywhere close to AGI and I don't think they ever will be (just a component). But that doesn't mean they aren't useful enough to chew up a lot of compute for the foreseeable future. Sure, but the question is how much compute? What if we're reaching the asymptotic limit of scaling (but still with some post-training and dataset curation gains to be had), and existing datacenters go from being used for training to inference instead. How long before another data center needs to be built or updated with latest NVIDIA GPUs? Is this (LLM-based AI) just a GPU-upgrade market, or still one growing explosively ?
danjl 2y ago
The reasons GPUs are useful is that they allow us to stay closer Moore's curve. Power is the problem, not AI algorithms. AI algorithms are a tensor processing best case right now. It took us decades to develop widely parallel algorithms for graphics rendering. AI will develop new algorithms. And algorithms will be optimized to be more sparse, which will make them less of an ideal case for the widely parallel hardware. But we will likely need both serial and widely parallel cores to handle any future algorithm.
janwas 2y ago
Brute force hardware locks us into the local minimum of matmul. Stella Nera demonstrated better power efficiency than GPUs, using approx matmul. Maybe the next generation of special purpose HW can help, too. But they come after 2-4 years. Who knows what we will want then? I'd rather have flexible HW.

HarHarVeryFunny 2y ago

> To be clear, I agree that LLMs are not anywhere close to AGI and I don't think they ever will be (just a component). But that doesn't mean they aren't useful enough to chew up a lot of compute for the foreseeable future. Sure, but the question is how much compute? What if we're reaching the asymptotic limit of scaling (but still with some post-training and dataset curation gains to be had), and existing datacenters go from being used for training to inference instead. How long before another data center needs to be built or updated with latest NVIDIA GPUs? Is this (LLM-based AI) just a GPU-upgrade market, or still one growing explosively ?

danjl 2y ago

The reasons GPUs are useful is that they allow us to stay closer Moore's curve. Power is the problem, not AI algorithms. AI algorithms are a tensor processing best case right now. It took us decades to develop widely parallel algorithms for graphics rendering. AI will develop new algorithms. And algorithms will be optimized to be more sparse, which will make them less of an ideal case for the widely parallel hardware. But we will likely need both serial and widely parallel cores to handle any future algorithm.

janwas 2y ago

Brute force hardware locks us into the local minimum of matmul. Stella Nera demonstrated better power efficiency than GPUs, using approx matmul. Maybe the next generation of special purpose HW can help, too. But they come after 2-4 years. Who knows what we will want then? I'd rather have flexible HW.