5 ms·

CPU cycles are limitless. Most of the time the CPU is waiting for the memory system.

by streetcat1 7y ago

zozbot234 7y ago
True enough, but modern software stacks also have a lot of wasteful memory use.
- streetcat1 7y ago
  Right. Hence Moore's law is irrelevant. I.e. the CPU is NOT the bottleneck. For example, who cares if you have 64 core , 5Ghz cpu, if the still doing I/O from an HDD.
dragontamer 7y ago
We're talking about 16-bit floats, or even INT8 or INT4, as the basis of neural nets these days, pushing the problem back to CPU-cycles. CNNs in particular recycle the same set of weights over-and-over again, fitting inside of the tiny caches (or shared-memory in GPUs), allowing for the compute-portion of the hardware to really work the data. > CPU cycles are limitless. Most of the time the CPU is waiting for the memory system. CPUs go out-of-order, deeply pipelines, and speculative so that they have work to do even while waiting for the memory system. The typical CPU has over 200+ instructions in flight in parallel these days. (200+ sized reorder buffers and "shadow registers" to support this hidden parallelism), and that's split between two threads for better efficiency ("Hyperthreading"). GPUs can have 8x warps / wavefronts per SM (NVidia) or CU (AMD) waiting for memory. If one warp/wavefront (a group of 32 or 64 threads) is waiting for memory, the GPU will switch to another "ready to run" warp/wavefront. It takes some programmer effort to understand this process and write high-performance code. But its doable with some practice.
- streetcat1 7y ago
  So I assume that we are talking about Moores law for cpu/gpu and not for memory. The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models. In addition, the problem with deep learning, in general, is the seq nature of the alg. I.e it is parallel within the layer, but not between layers. And for multi gpu setup, again it is the communication link between the GPUs. So I think that the nature of the current state of the art optimization alg are what matter.
  - dragontamer 7y ago
    > The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models. Well... everything "depends on the model". Some models will be GPU-compute limited, others will be bandwidth-limited, and others will be capacity limited. > In addition, the problem with deep learning, in general, is the seq nature of the alg. I.e it is parallel within the layer, but not between layers. And for multi gpu setup, again it is the communication link between the GPUs. You should increase the size of the model to increase parallelism. Any problem has sequential-bits and parallel bits, so we can't stop the sequential nature of problems. But the idea is to think NOT in terms of Ahmdal's law, but instead in terms of Gustafson's law. When you have a computer that's twice-as-parallel, you need to double the work done. You can't reasonably expect things to get faster-and-faster (Ahmdal's law). Instead, you load up more-and-more work the wider-and-wider machines get. In the case of neural nets, instead of doing 128x128x5 sized kernels, you upgrade to 256x256x5 sized kernels as GPUs get thicker. Moore's law was never about the "speed of computers", but instead about "the number of transistors". As such, its Gustafson's law that best scales with Moore's law. We've got another 5 to 10 years left in Moore's law by the way: denser memory, denser GPU compute, 5nm-class chips (probably networked as a bunch of chiplets). We will have more CPU / GPU power, as well as more RAM density, in the future. The question is how to organize our code so that we'll be ready 5 years from now for the next-gen of hardware.
    - streetcat1 7y ago
      Thank you for your insights. Good info.
  - solidasparagus 7y ago
    > The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models This is somewhat true, but not at clear cut as you make it seem. Most deep learning training I've been part of is realistically bottlenecked by compute. Increased memory usage is a way to improve the compute efficiency (less time moving between GPU and CPU). I haven't worked hands-on with the recent big NLP models so I could be mistaken in that domain, but there is a lot of evidence that model parallelism and hybrid parallelism work (technically - financially is less clear) so the memory limits of a single GPU/ASIC are less important than previously. One day I hope to see metaoptimization - for example particle swarm optimization - used to parallelize training so it is truly horizontally scalable, but right now the cost/hardware availability is a blocker.

CPU cycles are limitless. Most of the time the CPU is waiting for the memory system.

zozbot234 7y ago

True enough, but modern software stacks also have a lot of wasteful memory use.

streetcat1 7y ago
Right. Hence Moore's law is irrelevant. I.e. the CPU is NOT the bottleneck. For example, who cares if you have 64 core , 5Ghz cpu, if the still doing I/O from an HDD.

dragontamer 7y ago

We're talking about 16-bit floats, or even INT8 or INT4, as the basis of neural nets these days, pushing the problem back to CPU-cycles. CNNs in particular recycle the same set of weights over-and-over again, fitting inside of the tiny caches (or shared-memory in GPUs), allowing for the compute-portion of the hardware to really work the data. > CPU cycles are limitless. Most of the time the CPU is waiting for the memory system. CPUs go out-of-order, deeply pipelines, and speculative so that they have work to do even while waiting for the memory system. The typical CPU has over 200+ instructions in flight in parallel these days. (200+ sized reorder buffers and "shadow registers" to support this hidden parallelism), and that's split between two threads for better efficiency ("Hyperthreading"). GPUs can have 8x warps / wavefronts per SM (NVidia) or CU (AMD) waiting for memory. If one warp/wavefront (a group of 32 or 64 threads) is waiting for memory, the GPU will switch to another "ready to run" warp/wavefront. It takes some programmer effort to understand this process and write high-performance code. But its doable with some practice.

streetcat1 7y ago
So I assume that we are talking about Moores law for cpu/gpu and not for memory. The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models. In addition, the problem with deep learning, in general, is the seq nature of the alg. I.e it is parallel within the layer, but not between layers. And for multi gpu setup, again it is the communication link between the GPUs. So I think that the nature of the current state of the art optimization alg are what matter.
- dragontamer 7y ago
  > The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models. Well... everything "depends on the model". Some models will be GPU-compute limited, others will be bandwidth-limited, and others will be capacity limited. > In addition, the problem with deep learning, in general, is the seq nature of the alg. I.e it is parallel within the layer, but not between layers. And for multi gpu setup, again it is the communication link between the GPUs. You should increase the size of the model to increase parallelism. Any problem has sequential-bits and parallel bits, so we can't stop the sequential nature of problems. But the idea is to think NOT in terms of Ahmdal's law, but instead in terms of Gustafson's law. When you have a computer that's twice-as-parallel, you need to double the work done. You can't reasonably expect things to get faster-and-faster (Ahmdal's law). Instead, you load up more-and-more work the wider-and-wider machines get. In the case of neural nets, instead of doing 128x128x5 sized kernels, you upgrade to 256x256x5 sized kernels as GPUs get thicker. Moore's law was never about the "speed of computers", but instead about "the number of transistors". As such, its Gustafson's law that best scales with Moore's law. We've got another 5 to 10 years left in Moore's law by the way: denser memory, denser GPU compute, 5nm-class chips (probably networked as a bunch of chiplets). We will have more CPU / GPU power, as well as more RAM density, in the future. The question is how to organize our code so that we'll be ready 5 years from now for the next-gen of hardware.
  - streetcat1 7y ago
    Thank you for your insights. Good info.
- solidasparagus 7y ago
  > The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models This is somewhat true, but not at clear cut as you make it seem. Most deep learning training I've been part of is realistically bottlenecked by compute. Increased memory usage is a way to improve the compute efficiency (less time moving between GPU and CPU). I haven't worked hands-on with the recent big NLP models so I could be mistaken in that domain, but there is a lot of evidence that model parallelism and hybrid parallelism work (technically - financially is less clear) so the memory limits of a single GPU/ASIC are less important than previously. One day I hope to see metaoptimization - for example particle swarm optimization - used to parallelize training so it is truly horizontally scalable, but right now the cost/hardware availability is a blocker.