4 ms·

In my observations, GPUs are the way they are largely because Physics demands sacrifices if you want that kind of performance. Those same demands apply to CPUs

In my observations, GPUs are the way they are largely because Physics demands sacrifices if you want that kind of performance. Those same demands apply to CPUs, but Intel has been fighting against them because software engineers hate to change anything about how they code. Well... the fight was lost years ago and Intel is conceding to Physics by integrating architecture that was originally designed for a CPU-GPU hybrid. With AVX-512 and CUDA, Intel and Nvidia are trying to find the best developer experience they can to deal with the inevitable future of computing that is somewhere between those two experiences currently.

Const-me 6y ago
> inevitable future of computing that is somewhere between those two experiences currently. Not sure that ever happens. The two are very different in their performance characteristics. CPUs have to stay focused on single-threaded performance. Single threaded performance = minimizing latency, that’s why in CPUs so many transistors and electricity is spent on cache hierarchy and coherency protocol, instructions reordering, ILP, microops fusion, prefetcher, branch prediction incl. indirect one, and so on. GPUs don’t care about single-threaded performance, all they care is throughput on massively parallel tasks. They ignore memory latency, switching threads until data arrives, makes caches much simpler. They don’t need high clock frequencies (this alone improves their flops/watt by a large factor), don’t do branch prediction, memory model is less strict, and so on. That’s how they can spend that many transistors on FPUs and ALUs. P.S. Intel tried many-slow-cores with Larrabee / knights landing / etc., all they got was a very expensive lesson, people didn’t want that stuff. I’ve heard a rumor initially Sony wanted to use another Cell CPU for the GPU in PS3, again, didn’t worked well enough and they negotiated with NVidia instead.
- corysama 6y ago
  All true. Problem is that single-threaded latency minimization has been tapped out for a long time now. Physics says No. Memory is high-latency because of the speed of light. Larger caches have diminishing returns. Clocks can't go faster without melting the chips. There is no more prediction and pipelining to be dug out of a serial stream of instructions. We can still add more transistors. But, there is nowhere for them to contribute but in going wider. High latency, high throughput. High concurrency and careful management of the memory hierarchy are the only ways to pull that off. When you accept that fate, you start out with the Cell processor and you pass through Nvidia's Ampere chip --which has somewhat more thread independence and better automatic caching than past GPUs, but also finally gained manual async DMAs to local SRAM like the Cell had. Where we'll end up, I don't know.
  - Const-me 6y ago
    When I look at general-purpose single-threaded benchmarks, I see a good progress. I’m pretty sure we’ll see very good progress on that, because Intel/AMD competition, also because inexpensive SSDs removed the storage bottleneck. I agree on physical limitations, but the research still goes on, smart people still able to use these extra transistors in a way that’s useful for performance. Even for single-threaded one. Another thing, software is progressively better at using many cores. It’s much easier to move large pieces of code to another CPU thread (like some browsers do with JS compiler) keeping it sequential code, than it is to re-write code as a large set of small tasks / threads. Many useful algorithms can’t use fine-grained parallelism much, or at all, e.g. streaming stuff like gzip or parsers.
    - corysama 6y ago
      I really miss the 286->Pentium days when you had to buy a new machine every year because it would be literally 50-100% faster at everything. Every year! These days people call a 9% single-thread improvement a good year. https://www.cpubenchmark.net/year-on-year.html https://www.cpubenchmark.net/year-on-year.html https://www.karlrupp.net/wp-content/uploads/2018/02/42-years-processor-trend.png https://www.karlrupp.net/wp-content/uploads/2018/02/42-years... BTW: Thanks for http://const.me/articles/simd/simd.pdf http://const.me/articles/simd/simd.pdf I came across it recently and posted it to https://www.reddit.com/r/simd/ https://www.reddit.com/r/simd/
  - paulmd 6y ago
    > All true. Problem is that single-threaded latency minimization has been tapped out for a long time now. Physics says No. Memory is high-latency because of the speed of light. Larger caches have diminishing returns. Clocks can't go faster without melting the chips. There is no more prediction and pipelining to be dug out of a serial stream of instructions. Someone should probably tell AMD and Intel they need to stop, then, because Ice Lake was +18% IPC, Tiger Lake will likely be another 5-10%, and Zen3 will be around 15% higher IPC. What has truly stalled IPC is 10nm. Intel has had higher-IPC architectures for a long time, they simply couldn't manufacture them. And AMD was coming so far from behind that it took them 3 generations of Zen to even catch up to Intel. The last few years have been a rather unfortunate "CPU dark age" but things are finally breaking loose once again. Ice Lake launched 9 months ago, Intel backported Tiger Lake to 14nm and that will launch probably around Christmas or CES, Milan is launching probably within the next 6-8 weeks, and Tiger Lake is launching this month.
PaulHoule 6y ago
CUDA is night-and-day different from Intel's nightmare. CUDA in particular hides the exact width of the processing elements from the programmer, so you don't have to rewrite your CUDA applications to take advantage of the next generation of GPU the way you have to do with AVX1,2,3,... Mind you, NVIDIA's drivers compile your shaders to their proprietary instruction set, which is a path not taken in conventional CPU architecture even though it keeps coming up. (e.g. the belt)
- dragandj 6y ago
  FWIW, Intel supports OpenCL for their CPUs, so you can use similar general parallel programming code which is compiled down to AVX-512 (fairly) transparently.

Const-me 6y ago

> inevitable future of computing that is somewhere between those two experiences currently. Not sure that ever happens. The two are very different in their performance characteristics. CPUs have to stay focused on single-threaded performance. Single threaded performance = minimizing latency, that’s why in CPUs so many transistors and electricity is spent on cache hierarchy and coherency protocol, instructions reordering, ILP, microops fusion, prefetcher, branch prediction incl. indirect one, and so on. GPUs don’t care about single-threaded performance, all they care is throughput on massively parallel tasks. They ignore memory latency, switching threads until data arrives, makes caches much simpler. They don’t need high clock frequencies (this alone improves their flops/watt by a large factor), don’t do branch prediction, memory model is less strict, and so on. That’s how they can spend that many transistors on FPUs and ALUs. P.S. Intel tried many-slow-cores with Larrabee / knights landing / etc., all they got was a very expensive lesson, people didn’t want that stuff. I’ve heard a rumor initially Sony wanted to use another Cell CPU for the GPU in PS3, again, didn’t worked well enough and they negotiated with NVidia instead.

corysama 6y ago
All true. Problem is that single-threaded latency minimization has been tapped out for a long time now. Physics says No. Memory is high-latency because of the speed of light. Larger caches have diminishing returns. Clocks can't go faster without melting the chips. There is no more prediction and pipelining to be dug out of a serial stream of instructions. We can still add more transistors. But, there is nowhere for them to contribute but in going wider. High latency, high throughput. High concurrency and careful management of the memory hierarchy are the only ways to pull that off. When you accept that fate, you start out with the Cell processor and you pass through Nvidia's Ampere chip --which has somewhat more thread independence and better automatic caching than past GPUs, but also finally gained manual async DMAs to local SRAM like the Cell had. Where we'll end up, I don't know.
- Const-me 6y ago
  When I look at general-purpose single-threaded benchmarks, I see a good progress. I’m pretty sure we’ll see very good progress on that, because Intel/AMD competition, also because inexpensive SSDs removed the storage bottleneck. I agree on physical limitations, but the research still goes on, smart people still able to use these extra transistors in a way that’s useful for performance. Even for single-threaded one. Another thing, software is progressively better at using many cores. It’s much easier to move large pieces of code to another CPU thread (like some browsers do with JS compiler) keeping it sequential code, than it is to re-write code as a large set of small tasks / threads. Many useful algorithms can’t use fine-grained parallelism much, or at all, e.g. streaming stuff like gzip or parsers.
  - corysama 6y ago
    I really miss the 286->Pentium days when you had to buy a new machine every year because it would be literally 50-100% faster at everything. Every year! These days people call a 9% single-thread improvement a good year. https://www.cpubenchmark.net/year-on-year.html https://www.cpubenchmark.net/year-on-year.html https://www.karlrupp.net/wp-content/uploads/2018/02/42-years-processor-trend.png https://www.karlrupp.net/wp-content/uploads/2018/02/42-years... BTW: Thanks for http://const.me/articles/simd/simd.pdf http://const.me/articles/simd/simd.pdf I came across it recently and posted it to https://www.reddit.com/r/simd/ https://www.reddit.com/r/simd/
- paulmd 6y ago
  > All true. Problem is that single-threaded latency minimization has been tapped out for a long time now. Physics says No. Memory is high-latency because of the speed of light. Larger caches have diminishing returns. Clocks can't go faster without melting the chips. There is no more prediction and pipelining to be dug out of a serial stream of instructions. Someone should probably tell AMD and Intel they need to stop, then, because Ice Lake was +18% IPC, Tiger Lake will likely be another 5-10%, and Zen3 will be around 15% higher IPC. What has truly stalled IPC is 10nm. Intel has had higher-IPC architectures for a long time, they simply couldn't manufacture them. And AMD was coming so far from behind that it took them 3 generations of Zen to even catch up to Intel. The last few years have been a rather unfortunate "CPU dark age" but things are finally breaking loose once again. Ice Lake launched 9 months ago, Intel backported Tiger Lake to 14nm and that will launch probably around Christmas or CES, Milan is launching probably within the next 6-8 weeks, and Tiger Lake is launching this month.

PaulHoule 6y ago

CUDA is night-and-day different from Intel's nightmare. CUDA in particular hides the exact width of the processing elements from the programmer, so you don't have to rewrite your CUDA applications to take advantage of the next generation of GPU the way you have to do with AVX1,2,3,... Mind you, NVIDIA's drivers compile your shaders to their proprietary instruction set, which is a path not taken in conventional CPU architecture even though it keeps coming up. (e.g. the belt)

dragandj 6y ago
FWIW, Intel supports OpenCL for their CPUs, so you can use similar general parallel programming code which is compiled down to AVX-512 (fairly) transparently.