4 ms·

So I assume that we are talking about Moores law for cpu/gpu and not for memory. The bottleneck for GPU today is the amount of memory on board and not compute.

So I assume that we are talking about Moores law for cpu/gpu and not for memory. The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models. In addition, the problem with deep learning, in general, is the seq nature of the alg. I.e it is parallel within the layer, but not between layers. And for multi gpu setup, again it is the communication link between the GPUs. So I think that the nature of the current state of the art optimization alg are what matter.

dragontamer 7y ago
> The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models. Well... everything "depends on the model". Some models will be GPU-compute limited, others will be bandwidth-limited, and others will be capacity limited. > In addition, the problem with deep learning, in general, is the seq nature of the alg. I.e it is parallel within the layer, but not between layers. And for multi gpu setup, again it is the communication link between the GPUs. You should increase the size of the model to increase parallelism. Any problem has sequential-bits and parallel bits, so we can't stop the sequential nature of problems. But the idea is to think NOT in terms of Ahmdal's law, but instead in terms of Gustafson's law. When you have a computer that's twice-as-parallel, you need to double the work done. You can't reasonably expect things to get faster-and-faster (Ahmdal's law). Instead, you load up more-and-more work the wider-and-wider machines get. In the case of neural nets, instead of doing 128x128x5 sized kernels, you upgrade to 256x256x5 sized kernels as GPUs get thicker. Moore's law was never about the "speed of computers", but instead about "the number of transistors". As such, its Gustafson's law that best scales with Moore's law. We've got another 5 to 10 years left in Moore's law by the way: denser memory, denser GPU compute, 5nm-class chips (probably networked as a bunch of chiplets). We will have more CPU / GPU power, as well as more RAM density, in the future. The question is how to organize our code so that we'll be ready 5 years from now for the next-gen of hardware.
- streetcat1 7y ago
  Thank you for your insights. Good info.
solidasparagus 7y ago
> The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models This is somewhat true, but not at clear cut as you make it seem. Most deep learning training I've been part of is realistically bottlenecked by compute. Increased memory usage is a way to improve the compute efficiency (less time moving between GPU and CPU). I haven't worked hands-on with the recent big NLP models so I could be mistaken in that domain, but there is a lot of evidence that model parallelism and hybrid parallelism work (technically - financially is less clear) so the memory limits of a single GPU/ASIC are less important than previously. One day I hope to see metaoptimization - for example particle swarm optimization - used to parallelize training so it is truly horizontally scalable, but right now the cost/hardware availability is a blocker.

dragontamer 7y ago

> The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models. Well... everything "depends on the model". Some models will be GPU-compute limited, others will be bandwidth-limited, and others will be capacity limited. > In addition, the problem with deep learning, in general, is the seq nature of the alg. I.e it is parallel within the layer, but not between layers. And for multi gpu setup, again it is the communication link between the GPUs. You should increase the size of the model to increase parallelism. Any problem has sequential-bits and parallel bits, so we can't stop the sequential nature of problems. But the idea is to think NOT in terms of Ahmdal's law, but instead in terms of Gustafson's law. When you have a computer that's twice-as-parallel, you need to double the work done. You can't reasonably expect things to get faster-and-faster (Ahmdal's law). Instead, you load up more-and-more work the wider-and-wider machines get. In the case of neural nets, instead of doing 128x128x5 sized kernels, you upgrade to 256x256x5 sized kernels as GPUs get thicker. Moore's law was never about the "speed of computers", but instead about "the number of transistors". As such, its Gustafson's law that best scales with Moore's law. We've got another 5 to 10 years left in Moore's law by the way: denser memory, denser GPU compute, 5nm-class chips (probably networked as a bunch of chiplets). We will have more CPU / GPU power, as well as more RAM density, in the future. The question is how to organize our code so that we'll be ready 5 years from now for the next-gen of hardware.

streetcat1 7y ago
Thank you for your insights. Good info.

solidasparagus 7y ago

> The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models This is somewhat true, but not at clear cut as you make it seem. Most deep learning training I've been part of is realistically bottlenecked by compute. Increased memory usage is a way to improve the compute efficiency (less time moving between GPU and CPU). I haven't worked hands-on with the recent big NLP models so I could be mistaken in that domain, but there is a lot of evidence that model parallelism and hybrid parallelism work (technically - financially is less clear) so the memory limits of a single GPU/ASIC are less important than previously. One day I hope to see metaoptimization - for example particle swarm optimization - used to parallelize training so it is truly horizontally scalable, but right now the cost/hardware availability is a blocker.